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FOREWORD 


The  Army’s  proactive  effort  to  exploit  advanced  information  systems  requires  research  at 
the  frontiers  of  the  information  age.  Therefore,  the  Mounted  Battlespace  Battle  Lab  is 
conducting  a  series  of  Concept  Experimentation  Programs  (CEPs)  to  explore  advanced  system 
capabilities  and  training  requirements  in  the  2010-2015  timeframe.  To  support  this  CEP 
program,  the  U.S.  Army  Research  Institute  (ARI)  joined  this  battle  lab's  CEP  evaluation  team. 
This  team  helped  define  initial  research  issues  and  lead  evaluation  efforts  to  address  these  issues. 

Based  on  ARI’s  role  in  this  CEP,  this  report  describes  research  methods  and  results 
related  to  battle  command  information  requirements  and  the  military  decision  making  process. 
The  findings,  although  exploratory,  provide  a  benchmark  for  future  efforts  and  suggestions  for 
improving  advanced  information  systems.  Limitations  and  lessons  learned  on  research  methods 
are  considered.  Recommendations  for  improving  training  and  evaluation  methods  are  provided. 
These  recommendations  stress  the  impact  of  advanced  information  systems  on  the  process  of 
meeting  information  requirements  and  making  decisions.  Method  recommendations  emphasize 
that  advanced  information  systems  are  powerful  research  tools  to  help  meet  training  and 
evaluation  requirements. 

This  work  was  part  of  ARI's  research  program  to  train  the  force.  The  objective  of  ARI's 
Future  Battlefield  Conditions  (FBC)  team  at  Fort  Knox  is  to  enhance  soldier  preparedness 
through  development  of  training  and  evaluation  methods  to  meet  future  battlefield  conditions. 
This  report  represents  efforts  under  Work  Package  2228  Force  XXI  Training  Methods  and 
Strategies  (FASTTRAIN).  The  ARI's  research  on  training  requirements  and  evaluation  methods 
is  supported  by  a  Memorandum  of  Agreement  (MOA)  between  the  U.S.  Army  Armor  Center 
(USAARMC)  and  ARI.  This  MOA  is  titled  Manpower,  Personnel  and  Training  Research, 
Development,  Test,  and  Evaluation  for  the  Mounted  Forces,  16  October  1995. 


TAM.  SIMUTIS 
technical  Director 
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ASSESSING  BATTLE  COMMAND  INFORMATION  REQUIREMENTS  AND  THE 
MILITARY  DECISION  MAKING  PROCESS  IN  A  CONCEPT  EXPERIMENTATION 
PROGRAM 

EXECUTIVE  SUMMARY 


Research  Requirement: 

Research  is  needed  to  investigate  concepts  for  re-engineering  of  the  Tactical  Operations 
Center  (TOC)  at  battalion  and  brigade  level,  in  order  to  take  full  advantage  of  information  age 
technology.  Recent  Army  Warfighting  Experiments  (AWEs)  and  other  investigations  have 
documented  shortcomings  in  the  Army’s  current  command,  control,  communication,  computer 
and  intelligence  (C4I)  systems.  The  Mounted  Battlespace  Battle  Lab  (MBBL)  is  conducting  a 
series  of  Concept  Experimentation  Programs  (CEPs)  to  explore  new  C4I  systems,  organization 
and  processes  for  the  future  battalion  and  brigade  TOCs.  The  research  described  in  this  report 
used  the  first  CEP  as  a  vehicle  to  address  two  issues  concerning  the  re-engineered  TOC.  The 
first  issue,  which  related  to  new  C4I  tools,  was:  How  will  a  future  maneuver  battle  commander’s 
digital  display  meet  the  mission,  enemy,  terrain,  troops  and  time  (METT-T)  information 
requirements  of  the  commander?  The  second  issue,  which  concerned  potential  changes  in  staff 
processes,  was:  Given  the  future  battle  commander’s  display  capabilities,  what  are  the  changes 
to  the  decision  making  process? 

Procedure: 

Data  collection  concerning  the  two  issues  described  above  was  integrated  into  the 
conduct  of  the  CEP.  Immediately  after  each  of  the  three  test  missions,  test  players  and  observers 
completed  surveys  designed  to  address  adequacy  of  C4I  systems  to  meet  information 
requirements  and  changes  in  the  decision  making  process.  Comments  in  hot  washes  were  also 
used  to  address  the  two  issues. 

Findings: 

Results  provided  information  addressing  both  issues,  that  is,  how  well  the  C4I  systems 
met  staff  information  requirements  and  potential  changes  in  the  decision  making  process. 
Overall,  the  results  indicated  that  test  participants’  information  requirements  were  met  or  nearly 
met.  Notably,  none  of  the  information  components  rated  during  the  evaluation  approached  the 
level  of  “Exceeded.”  These  results  suggest  there  is  a  substantial  need  for  C4I  system 
improvements  to  better  meet  information  requirements.  Concerning  the  second  issue,  there  were 
few  major  changes  reported  by  respondents  concerning  the  military  decision  making  process 
(MDMP).  Overall,  both  test  players  and  observers  rated  most  sub-steps  as  “Unchanged.” 
However,  test  players  appeared  more  likely  to  rate  MDMP  steps  as  “Changed”  than  were 
observers.  This  finding,  along  with  the  questionnaire  comments,  suggested  minimal  changes  in 
the  MDMP  during  the  CEP.  Many  questionnaire  comments  stress  that  the  MDMP  steps  were 
performed  in  a  less  formal,  more  streamlined  manner.  Nonetheless,  the  steps  were  still 
performed.  These  findings  are  similar  to  those  reported  by  Elliott,  Sanders  and  Quinkert  (1996) 
concerning  training  lessons  learned  from  the  Army  Warfighting  Experiment  Focused  Dispatch. 
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During  Focused  Dispatch,  conventional  doctrinal  materials  for  the  MDMP  were  available,  but 
not  their  digital  counterparts.  The  Task  Force  personnel  in  the  CEP,  as  in  Focused  Dispatch, 
were  creating  new  decision  making  tactics,  techniques  and  procedures  (TTP)  during  the 
evaluation.  In  the  absence  of  new  doctrine,  participants  fall  back  on  old  TTP  and  doctrine. 
Therefore,  the  lack  of  dramatic  change  in  the  MDMP  during  the  CEP  was  not  surprising. 

Utilization  of  Findings: 

The  methods  and  results  reported  for  this  initial  CEP  provide  a  preliminary  baseline  for 
assessing  information  requirements  and  military  decision  making  in  future  operations.  The 
results  are  in  no  way  definitive,  but  they  provide  a  benchmark  for  future  CEPs  and  related 
efforts.  The  results  also  provide  many  valuable  user-based  suggestions  for  both  C4I  system  and 
evaluation  improvements.  Specific  limitations  to  the  manual  measures  developed  and  used  to 
assess  information  requirements  and  military  decision  making  are  also  described. 
Recommendations  to  improve  these  measures  are  also  addressed. 
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Assessing  Battle  Command  Information  Requirements  and  the  Military  Decision 
Making  Process  in  a  Concept  Experimentation  Program 

Introduction 

This  report  describes  research  conducted  to  assess  information  requirements  and  military 
decision  making  as  a  result  of  advanced  organizations  and  equipment  for  battle  command.  The 
purpose  of  introducing  these  innovations  was  to  enable  command  and  staff  personnel  from  a 
battalion  task  force  to  take  advantage  of  the  information  processing  capabilities  anticipated  in  the 
2010-2015  timeframe.  This  research  was  undertaken  as  a  part  of  a  reengineered  battalion  tactical 
operations  center  (TOC)  concept  experimentation  program  (CEP),  conducted  by  the  Mounted 
Battlespace  Battle  Lab  (MBBL)  at  Fort  Knox,  Kentucky.  Here,  only  a  brief  overview  of  the 
background  and  method  for  this  CEP  is  provided,  as  more  complete  reports  are  available.1  After 
this  overview,  this  report  focuses  on  the  methods  used  during  this  CEP  to  assess  battle  command 
information  requirements  and  the  military  decision  making  process  (MDMP),  and  the  results  and 
recommendations  for  method  improvements.  This  report  focuses  on  lessons  learned  for  training 
and  particularly  evaluating  information  age  forces. 

Army  Need 

Although  the  Army  has  made  substantial  advances  in  incorporating  digital  technology 
into  command  and  control  systems  in  recent  years,  results  of  numerous  simulations  and  field 
tests,  including  Advanced  Warfighting  Experiments  (AWEs),  show  several  deficiencies 
(USAARMC,  1998).  First,  individual  digital  systems  (referred  to  as  Command,  Control, 
Communication,  Computer  and  Intelligence  [C4I]  systems,  in  this  report)  do  not  completely  meet 
users’  needs.  As  one  example,  the  All  Source  Analysis  System  (ASAS)  receives  digital  spot 
reports  on  enemy  vehicle  locations  from  platoon  wingmen  throughout  the  task  force.  The 
intelligence  officer  (S2)  should  then  manually  integrate  or  fuse  these  reports,  and  eliminate 
redundant  information  and  icons.  Thus  while  some  low-level  tasks  of  the  intelligence  section  are 
automated  by  digital  systems,  others  still  remain,  and  new  ones  are  created.  Second,  the  Army’s 
current  C4I  systems  such  as  the  Army  Tactical  Command  and  Control  System  (ATCCS),  are 
stovepiped  or  incompatible  in  that  they  communicate  primarily  within  their  own  battlefield 
operating  system  (BOS).  System  incompatibility  makes  it  difficult,  for  example,  to  portray  a 
common  picture  of  the  battlefield  situation  on  any  one  system  or  across  systems.  Finally,  the 
current  organization  and  operating  procedures  of  a  battalion  staff  may  not  be  optimal  for  digital 
systems.  Current  organization  is  based  on  staff  sections  organized  around  a  BOS.  The 
assumption  is  that  each  section  develops  a  separate  piece  of  the  puzzle,  rather  than  all  viewing  an 
assembled  puzzle. 

The  premise  of  the  Army’s  recent  series  of  AWEs  was  that  advances  in  information 
technology  have  created  the  potential  for  a  revolution  in  military  affairs.  Just  as  new  weapon 
systems  may  dramatically  change  how  we  fight,  information  technology  advances  require  an 


1  For  a  more  detailed  discussion  of  the  CEP  design  and  methods  see  Elliott,  Sterling  and  Lickteig  (1998)  and 
USAARMC  (1998).  For  a  more  detailed  discussion  of  the  C’l  systems,  also  referred  to  as  “tools,”  see  the  User’s 
Manual  (USAARMC,  1997). 
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equal  examination.  The  Army,  to  date,  has  only  automated  its  current  organization  and 
procedures.  However,  information  technology  changes  the  fundamental  assumptions  that 
underlie  current  Army  operations  and  why  the  Army  operates  as  it  does.  The  Army  should 
understand  and  prepare  for  future  technology  today  if  it  is  to  utilize  it  tomorrow. 

The  Army  needs  to  develop  and  implement  lessons  learned  about  C4I  systems  to  improve 
training  and  force  development.  Soldier-based  research,  such  as  the  AWEs  and  CEPs,  provides  a 
venue  for  identifying,  implementing,  and  refining  lessons  learned.  For  example,  current 
procedures  emphasize  a  lock-step  method  of  commander,  staff,  and  subordinate  unit  leaders  all 
doing  their  tasks  in  a  linear  versus  collaborative  manner.  Future  staffs  with  C4I  tools  will  likely 
need  a  different,  more  rapid  decision  making  procedure  in  order  to  take  full  advantage  of  the 
information  dominance  these  tools  will  provide. 

MBBL  Approach 

Numerous  AWEs  have  documented  deficiencies  in  legacy  C4I  systems.  Identified 
deficiencies  include  shortcomings  in  meeting  user  information  requirements  and  supporting  the 
military  decision  making  process.  This  CEP,  therefore,  was  based  upon  the  premise  that  future 
technology  would  enable  the  creation  of  systems  in  which  currently  documented  deficiencies  in 
legacy  C4I  systems  would  be  substantially  corrected.  Thus,  the  MBBL  directed  its  efforts  to 
exploring  issues  arising  when  the  currently  identified  C4I  deficiencies  were  corrected. 
Accordingly,  the  initial  CEPs  have  at  least  two  main  objectives.  First,  to  assess  how  well  these 
improved  C4I  systems  have  addressed  these  deficiencies.  Second,  if  these  identified  C4I 
deficiencies  are  corrected,  how  would  this  affect  the  future  commander  and  staff? 

To  achieve  the  capabilities  anticipated  with  advanced  C4I  systems  and  prepare  the  Army 
future  force  operations,  the  MBBL  initiated  a  series  of  CEPs  to  examine  the  effects  of  advanced 
digitization  on  battle  command  at  brigade  and  below.  As  defined,  battle  command  has  three 
main  elements.  “Battle  command  is  the  art  of  battle  decision-making,  leading  and  motivating 
soldiers  and  their  organizations  into  action  to  accomplish  missions  at  least  cost  to  soldiers  and  to 
the  nation”  (USAARMC,  1998,  p.  2).  The  MBBL  plan  for  these  CEPs,  called  the  Battle 
Command  Futures  Program,  is  to  conduct  a  CEP  approximately  every  six  months  for  three  years. 
These  CEPs  begin  with  a  focus  on  battle  command  at  the  battalion  task  force  level,  and  then 
progress  to  brigade.  The  objective  of  this  program  “is  to  develop  an  optimal  battle-command 
solution  for  future  commanders  at  the  brigade  level  and  below”  (K.  J.  Gunzelman,  personal 
communication,  December  1,  1997).  The  approach  requires  the  development  and  integration  of 
futuristic  technologies  and  investigation  of  the  battle  command  staff  organization  that  best 
exploits  the  capabilities  provided. 

The  MBBL  designed  and  executed  the  CEP.  The  Director  of  the  MBBL  made  a  decided 
commitment  to  advance  the  Army’s  investment  in  digital  technologies  by  proactively  exploring 
the  potential  of  technology  rather  than  retroactively  documenting  its  limitations.  The  AWEs  had 
repeatedly  demonstrated  that  current,  and  therefore  legacy,  C4I  systems  do  not  fully  meet  the 
Army’s  operational  requirements.  The  MBBL’s  effort  to  develop  an  advanced  C4I  system  for 
lower  echelon  battle  command  resulted  in  some  unprecedented  capabilities  and  the  opportunity 
for  military  users  to  directly  and  repeatedly  assess  their  value  in  virtually  simulated  missions. 
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Lessons  learned  from  this  CEP  will  inform  future  Army  efforts,  including  training,  doctrine,  and 
materiel  requirements,  as  well  as  the  MBBL’s  programmatic  series  of  CEPs,  and  iteratively  build 
toward  an  optimal  battle  command  solution. 

To  support  MBBL’s  efforts,  the  Armored  Forces  Research  Unit  (AFRU)  of  the  U.S. 
Army  Research  Institute  (ARI)  joined  the  CEP  evaluation  team.  The  independent  evaluation 
team  was  headed  by  the  Test  and  Evaluation  Coordination  (TECO)  Office  at  Fort  Knox, 
Kentucky  that  is  part  of  the  U.S.  Army  Operational  Test  and  Evaluations  Command.  The  team 
assisted  the  MBBL  in  defining  CEP  research  issues,  and  leading  evaluation  efforts  to  address 
these  issues.  Because  of  the  resource  and  design  limitations  for  this  initial  CEP,  to  include 
technical  and  control  group  shortcomings,  the  MBBL  and  the  evaluation  team  concurred  that  the 
CEP  was  exploratory  and  the  results  preliminary.  At  TECO’s  request,  ARI  assumed  primary 
responsibility  for  assessing  the  MBBL's  CEP  issues  on  battle  command  information 
requirements  and  military  decision  making.  Another  issue  involving  new  skills,  education  and 
experience  needed  by  future  staff  members  was  addressed  by  TECO  alone.  An  independent 
report  on  the  CEP  (Elliott  et  al.,  1998)  was  also  published  by  TECO). 

Methods 

Method  Overview  for  the  CEP 

This  section  provides  an  overview  of  the  method  for  the  first  CEP  conducted  in 
December  1997  in  the  Mounted  Warfare  Test  Bed  at  Fort  Knox. 

Participants 

The  participants  were  divided  into  two  groups:  test  players  and  observers.  The  test 
players  were  20  soldiers  including  a  Lieutenant  Colonel  as  the  Task  Force  commander  and  two 
Majors  as  the  executive  officer  and  operations  officer  who  were  all  from  the  16th  Cavalry 
Regiment  and  graduates  of  the  Command  and  General  Staff  College  (CGSC).  Staff  officers 
were  recent  Armor  Officer  Advanced  Course  (AOAC)  graduates.  Noncommissioned  officers 
(NCOs)  were  from  the  16th  Cavalry  Regiment  or  the  1st  Armor  Training  Brigade.  Of  the  14 
officers,  13  were  Armor  Specialty  (Branch  Code  12)  and  one  was  Infantry  (Branch  Code  11) . 

Of  the  six  NCOs,  four  were  military  occupational  specialty  (MOS)  19K  and  two  were  19D. 
Notable  features  are  that  these  test  players  represented  an  ad  hoc  versus  an  intact  command  and 
staff  organization,  and  that  only  the  commander  had  extended  experience  with  digital  C4I 
systems. 

The  second  group  of  participants,  referred  to  as  observers,  were  10  soldiers  from  the  16th 
Cavalry  Regiment  Observer/Controller  (O/C)  team  who  also  provided  data  for  this  experiment. 
During  the  CEP,  these  observers  monitored  test  player  performance  and  served  as  both  data 
collectors  and  data  providers.  The  observer  group  included  two  Majors,  five  Captains,  and  three 
NCOs.  The  Majors  headed  the  observer  team  and  led  the  hot  washes  after  each  mission. 
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Organization 


In  an  attempt  to  take  full  advantage  of  the  advanced  C4I  systems  simulated  in  the  CEP,  a 
new  battalion  staff  organization  was  implemented.  The  new  organization  was  similar  to  an 
organization  that  evolved  in  earlier  work  on  C4I  systems  by  ARI  (Leibrecht,  Meade,  Schmidt, 
Doherty  &  Lickteig,  1994).  This  re-engineered  TOC  structure  is  depicted  in  Figure  1.  In 
contrast,  a  battalion  TOC  traditionally  includes  a  main  command  post  (CP)  and  combat  trains 
command  post  (CTCP).  The  main  CP  has,  at  least,  BOS  representation  for  intelligence, 
operations,  fire  support,  and  mobility-countermobility.  The  CTCP  handles  the  personnel  and 
logistics  functions  for  the  task  force.  However,  the  re-engineered  TOC  consisted  of  four 
sections,  referred  to  as  nodes.  These  were  the  battalion  commander’s  vehicle,  a  jump  CP  (with 
the  executive  officer  [XO]),  a  current  operations  vehicle  and  a  future  plans  vehicle.  Each  node 
had  four  soldiers.  Except  for  the  traditional  roles  of  commander,  XO  and  operations  officer,  the 
participants  assumed  more  generic  roles  of  enemy  operations  officer,  friendly  operations  officer 
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Figure  1 .  Overview  of  organization  and  equipment  for  the  CEP. 

or  NCO,  plans  NCO  and  sensor  NCO.  Except  for  the  battalion  commander’s  battle  command 
vehicle  (BCV),  which  contained  three  officers  and  one  NCO,  the  other  nodes  contained  two 
officers  and  two  NCOs.  Overall,  there  were  nine  officers  and  seven  NCOs. 

Equipment 

The  suite  of  C4I  equipment  for  each  of  the  four  TOC  nodes  was  identical.  Figure  2 
provides  an  overview  illustration  of  how  this  equipment  was  configured  in  each  node  during  the 
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CEP.  For  each  node,  separate  monitors  for  the  friendly  operations  officer,  enemy  operations 
officer  and  officer  in  charge  (OIC)  provided  the  “common  picture.”  These  monitors  could  depict 
either  a  two-dimensional  plan  view  display  or  three-dimensional  stealth  view.  In  addition,  the 
node  OIC  had  a  large  screen  monitor  on  which  to  display  a  stealth  view,  as  well  as  a  whiteboard. 
The  unmanned  aerial  vehicle  (UAV)  sensor  display  was  used  by  the  enemy  operations  officer  in 
each  node.  A  brief  discussion  of  each  equipment  item,  or  tool,  is  provided  below.  The  Mounted 
Warfare  Test  Bed  (MWTB)  in  which  the  CEP  was  conducted  is  a  distributed  interactive 
simulation  (DIS)  facility.  It  employs  networked,  selective  fidelity  simulators  for  manned  entities 
and  modular  semi-automated  forces  (ModSAF)  for  unmanned  entities.  It  is  a  soldier-in-the-loop 
environment,  capable  of  crew  through  battalion-level,  full  mission  simulation  of  realistic  combat 
environments. 

ModSAF.  The  ModSAF  software  was  used  to  drive  the  DIS  used  in  this  CEP.  The 
ModSAF  simulates  and  controls  entities  that  exist  on  a  virtual  battlefield.  The  ModSAF  forces 
represent  unmanned  friendly  forces  and  opposing  force  (OPFOR)  entities.  These  entities  behave 
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C--UAV  Sensor  Display  D-Whiteboard  &  VTC 

Figure  2.  Overview  of  equipment  for  a  node. 

Note.  UAV  =  unmanned  aerial  vehicle;  VTC  =  video  teleconference. 

(i.e.  move,  shoot  and  communicate)  autonomously,  but  remain  under  operator  control  (hence 
semi-automated),  and  interact  over  a  network  with  each  other  and  with  manned  simulators.  The 
ModSAF  was  used  to  create  many  of  the  tool  capabilities  described  below. 

Plan  View  Display  (PVD).  The  PVD,  driven  by  ModSAF,  generated  and  maintained 
situational  information  in  the  form  of  an  electronic,  two-dimensional  (D)  situation  map 
(SITMAP).  Key  types  of  situational  information  routinely  updated  on  this  display  included  the 
location,  identity  and  status  of  all  friendly  entities,  and  the  location  and  identity  of  all  sensed 
opposing  entities.  The  PVD  was  designed  to  simulate  correction  of  many  of  the  identified 
deficiencies  in  the  current,  legacy  C4I  systems.  All  friendly  entities  were  represented,  with 
positions  continuously  updated.  For  enemy  entities,  the  PVD  represented  a  "fused  sensor"  view 
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of  the  battlefield.  All  enemy  entities  within  current  range  of  any  sensor  (e.g.,  Joint  Surveillance 
Target  Attack  Radar  System  [JSTARS],  UAV,  friendly  battlefield  platform)  appeared  on  the 
PVD.  However,  these  enemy  entities  were  continuously  updated  only  when  within  range  of  a 
sensor;  otherwise  the  entity  faded  and  eventually  disappeared.  Thus  the  PVD  did  not  present  an 
"all  seeing"  eye  view  of  enemy  activity,  but  only  those  enemy  entities  that  were  sensed.  The 
PVD  provided  no  interpretation  of  enemy  force  structure  (e.g.,  Advanced  Guard  Main  Body)  but 
would  aggregate  enemy  icons  into  units  (e.g.,  platoon,  company)  using  decision  rules  based  on 
relative  location  of  enemy  icons.  The  PVD  depicted  terrain  information  drawn  from  a  digital 
terrain  database  and  provided  various  map  tool  features  such  as  terrain  analysis  for  line-of-sight 
determinations  and  a  wide  range  of  selectable  map  features  (e.g.,  contours,  grid  lines,  and 
buildings).  The  PVD  also  supported  the  creation  and  depiction  of  operational  overlays  based  on 
a  selectable  set  of  control  measures.  However,  no  time  estimates  (e.g.,  time  for  friendly  or 
enemy  forces  to  traverse  a  certain  distance)  were  available  from  any  C4I  tools. 

The  PVD  served  as  the  primary  C4I  display  during  the  CEP  and  represented  the  future 
battle  command  display.  The  PVD  provided  the  following  functions:  high  resolution  color 
display,  relevant  common  picture,  line  of  sight  planning  tool,  stackable  overlays,  and  fire  support 
planning  and  execution  capability.  Detailed  description  of  the  capabilities  of  the  tools  associated 
with  the  PVD  (and  C4I  tools  in  general)  are  beyond  the  scope  of  this  report,  but  are  included  in 
the  user’s  manual  (USAARMC,  1997). 

Stealth  Display.  The  stealth  display  provided  test  players  a  three-dimensional  perspective 
view  of  the  simulated  battlefield.  The  situational  information  provided  by  the  PVD  and  stealth 
display  were  identical;  any  enemy  or  friendly  vehicles  appearing  on  the  PVD  could  also  be  seen 
on  the  stealth  display.  With  the  use  of  a  “space  ball,”  the  officer  in  charge  of  each  node  could 
“fly  over”  the  battlefield  to  visualize  how  both  the  friendly  and  enemy  forces  were  arrayed  on 
three-dimensional  terrain. 

UAV.  A  UAV  in  each  node  replicated  the  capability  of  obtaining  information  from  a 
multitude  of  sensors,  including  unmanned  ground  sensors.  The  UAV  was  designed  to  assist  with 
enemy  reconnaissance  and  terrain  analysis,  in  particular. 

Whiteboard.  The  Whiteboard  provided  the  officer  in  charge  of  each  node  the  capability 
to  copy  all  or  part  of  the  PVD  screen  at  any  time  and  draw  graphics  on  the  copy  with  the  lite  pen 
(mouse).  For  example,  an  operational  overlay,  course  of  action  (CO A)  or  operations  order 
(OPORD)  could  be  sketched  out  on  the  Whiteboard  and  transmitted  to  any  other  node.  The 
Whiteboard  could  be  used  in  conjunction  with  simulated  radio  and  video  teleconference  (VTC) 
capabilities  to  conduct,  for  example,  an  orders  brief  or  rehearse  a  mission. 

VTC  capability.  The  video  teleconference  capability  allowed  the  commander  and  his 
staff  to  see  and  talk  to  each  other  on  a  flat  screen  display.  It  gave  them  a  “face-to-face” 
capability  to,  for  example,  issue  commander’s  guidance  or  receive  input  from  the  staff. 

Electronic  mail  (e-mail).  Nodes  could  send  either  free-text  messages  or  overlays  created 
on  the  Whiteboard  to  other  nodes,  subordinates  (company  commanders)  or  higher  headquarters 
(role  players  for  brigade  staff). 
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Training 


Test  player  training  was  conducted  in  the  MWTB  facility,  in  classrooms  and  bay  areas. 
Table  1  depicts  the  schedule  and  types  of  training  conducted.  This  training  began  with  about  two 
days  of  individual  training  on  how  to  use  the  tools  for  their  duty  position  and  node.  Although 
some  individual  training  was  conducted  in  MWTB  in  classrooms,  the  majority  was  hands-on 
training  with  the  actual  systems  used  in  the  experiment.  Individual  training  was  primarily 
provided  by  a  group  of  personnel  called  research  assistants.  Next,  small  group  and  collective 
training  occurred  during  approximately  four  days  in  which  participants  conducted  a  training 
mission  and  pilot  mission.  Research  assistants  were  available  throughout  the  training  and  trial 
missions  to  assist  with  problems  encountered  using  the  C4I  tools. 

Observer  training  was  conducted  at  the  same  time  as  test  player  training.  Generally,  the 
observer  training  consisted  of  monitoring  test  player  training,  although  observers  received  hands- 
on  training  by  the  research  assistants  on  the  ModSAF  and  system  tools  on  2  December.  Training 
for  data  collectors,  observers,  and  members  of  the  evaluation  team  was  primarily  provided  in  the 
form  of  a  data  collection  rehearsal  conducted  after  the  pilot  mission.  Data  management 
reduction  personnel  were  trained  during  the  test  player  and  observer  collective  training  period. 

Evaluation  Instruments 


A  demographic  questionnaire  was  administered  to  the  test  players  and  observers  to  obtain 
information  about  prior  military  experience,  including  experience  with  digital  information 
systems  (Elliott  et  al.,  1998).  Four  manual  data  collection  instruments  were  used  to  address  CEP 
research  issues.  Two  of  these  instruments,  Battlefield  Visualization  Tools  and  TOC  Skills, 
Training,  and  Experience,  are  described  and  provided  in  Elliott  et  al.  An  Information 
Requirements  Questionnaire  (Appendix  A)  was  used  to  assess  battle  command  information 
requirements.  An  MDMP  Questionnaire  (Appendix  B)  was  used  to  assess  decision  making. 
These  questionnaires  are  both  described  later  in  this  report.  Other  evaluation  instruments  used 
during  the  CEP,  but  not  directly  related  to  CEP  research  issues,  are  described  in  Elliott  et  al.  In 
addition,  manual  and  electronic  recordings  were  made  of  test  player  comments  during  the  CEP 
hot  washes.  These  hot  wash  comments  were  also  used  to  address  both  the  research  issues 
discussed  in  this  report . 

Procedure 


As  indicated  in  Table  1,  the  CEP  began  with  an  overview  introduction  in  a  classroom 
followed  by  a  tour  of  the  simulation  bay’s  equipment  and  organizational  configuration.  Test 
players  then  received  approximately  two  days  of  training  on  how  to  use  the  individual  tools  and 
spent  four  days  conducting  a  training  mission  followed  by  pilot  mission.  All  training  and  test 
missions  were  conducted  on  a  terrain  database  for  the  National  Training  Center  (NTC).  The 
three  test  missions  were  movement  to  contact,  deliberate  defense  and  deliberate  attack  missions. 
During  all  missions  a  brigade  cell  monitored  and  coordinated  the  simulated  exercise  and 
provided  instructions  to  ModSAF  controllers  for  unmanned  friendly  and  all  opposing  force 
entities. 
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Table  1 


Training  Schedule 


Date 

Time 

Training 

1  Dec 

TOC  Experiment  Introduction 

0900-1000 

Commander’s  Time 

■iiiiimihM 

Walk-through  tour  and  ModSAF  overview 

Individual  training  with  ModSAF  tools  at  work  stations 

1500-  1600 

One-on-one  refresher  training;  individual  training  need  identified 

1600-1700 

Hot  wash  (review  of  day  events  and  lessons  learned) 

2  Dec 

0800-  1100 

ModSAF/Tools  -  Individual  skills  training 

1100-1200 

Node  internal  review/Hot  wash/Company  Commander  work  station 
training  observer  training 

1300-  1400 

Video:  DARPA  TOC  of  the  Future 

1400  -  1500 

Tactical  Decision  Making  Process;  TOC  Division  of  Labor  (A  Method) 

Individual  skills  verification/remediation 

mwmvbm 

Hot  wash  (Review/Lessons  Learned/Data  Instrument  Overview) 

3  Dec 

0800-  1600 

Capstone  TOC  Collective  Training  Exercise  (Issue  Attack  Mission 
OPORD) 

1600  -  1700 

Hot  wash  (Army  Research  Lab  Data  Instrument  Overview) 

4  Dec 

0800-  1600 

Continue  Capstone  Attack  Mission  Exercise  (Plan/Prepare  training) 

1600-  i  700 

Hot  wash  (Task  Force  Commander’s  Lessons  Learned) 

8  Dec 

0800  -  1400 

Continue  Capstone  Attack  Mission  Exercise 

Hot  wash  (includes  instrument  data  collection  practice) 

9  Dec 

MMW.frlflU 

Continue  Capstone  Attack  Mission  Exercise 

1000-1030 

Battlefield  Visualization  Class 

10  Dec 

Continue  Capstone  Attack  Mission  Exercise 

Hot  wash  (Lessons  Learned) 

Note.  DARPA  =  Defense  Advanced  Research  Projects  Agency. 


For  all  missions,  the  test  players  initially  received  a  digital  operations  order  from  brigade, 
and  then  were  required  to  plan,  prepare  and  execute  the  battalion  mission  using  the  tools  and 
organization  previously  described.  Test  missions  lasted  eight  to  nine  hours  from  issuance  of 
brigade  OPORD  until  end  exercise  was  declared  or  enemy  forces  were  destroyed.  Hot  washes 
were  held  after  each  mission  to  address  use  of  the  equipment,  training  and  information 
requirements,  decision  making,  standing  operating  procedures  (SOPs)  and  tactics,  techniques  and 
procedures  (TTPs). 

Manual  evaluation  instruments  were  administered  after  each  test  mission  and  at  the 
conclusion  of  the  CEP.  The  two  evaluation  instruments  central  to  this  report,  the  Information 
Requirements  (IR)  and  MDMP  Questionnaires,  were  only  administered  after  each  test  mission, 
prior  to  the  hot  wash.  The  MDMP  Questionnaire  was  provided  to  test  players  and  observers  in 
the  simulation  bay  at  their  assigned  duty  stations.  A  member  of  the  evaluation  team  distributed 
the  questionnaire  at  each  node  and  answered  any  questions  about  the  instrument.  The  IR 
Questionnaire  was  provided  to  test  players  and  observers  in  the  same  classroom  used  to  conduct 
hot  washes.  A  member  of  the  evaluation  team  distributed  the  questionnaire  to  each  participant 
and  answered  any  questions  about  the  instrument.  Quality  control  procedures  included  verifying 
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completion  of  information,  reviewing  for  interpretation  of  participants’  handwritten  information, 
and  verifying  the  questionnaire  corrections  were  completed.  After  data  entry  was  completed, 
printouts  of  the  database  were  made  and  verified  and  corrections  made. 

CEP  Research  Issues 


The  TOC  re-engineering  CEP  was  focused  on  three  research  issues  (Elliott  et  al.,  1998) . 
The  first  two  issues  centered  on  the  battle  commander’s  digital  display,  and  the  third  issue 
addressed  the  skills,  education  and  experience  required  for  future  command  and  staff  personnel. 
Specifically,  the  issues  were: 

Issue  1.  How  will  the  future  maneuver  battle  commander’s  digital  display  meet  the  mission, 
enemy,  terrain,  troops  and  time  (METT-T)  information  requirements  of  the  commander? 

Issue  2.  Given  the  future  battle  commander’s  display  capabilities,  what  are  the  changes  to  the 
decision  making  process? 

Issue  3.  What  skills,  education  and  experience  do  commanders  and  staff  need  for  future 
operations? 

This  report  addresses  only  the  first  two  CEP  issues;  for  information  related  to  the  third 
issue  see  Elliott  et  al.  (1998).  While  the  focal  point  of  issues  1  and  2  is  literally  the  battle 
commander’s  digital  display,  the  methods  used  to  address  these  issues  included  the  battle 
command  displays  and  the  entire  suite  of  C4I  equipment  used  during  the  CEP.  Advantages  and 
limitations  to  this  approach  are  considered  in  the  discussion  section.  Given  this  overview  of  the 
CEP,  this  report  now  describes  the  methods  used  to  assess  battle  command  information 
requirements  and  the  MDMP  during  the  CEP. 

Method  for  Assessing  Battle  Command  Information  Requirements 

The  lead  research  issue  for  this  CEP  was  to  determine  how  the  future  maneuver  battle 
commander’s  display  met  the  METT-T  information  requirements  of  the  commander.  Two 
methods  were  used  to  address  this  issue.  First,  an  IR  Questionnaire  was  developed  to  identify 
what  information  was  required  and  how  well  information  requirements  were  met.  This 
questionnaire  was  administered  to  all  participants  (test  players  and  observers)  at  the  end  of  each 
test  mission.  Second,  test  player  comments  made  during  hot  wash  discussions  were  recorded 
manually  and  electronically. 

Instruments 


The  IR  Questionnaire  is  provided  in  Appendix  A.  This  questionnaire  used  closed-  and 
open-ended  approaches  to  identify  component  types  of  information  required  by  the  participants’ 
duty  position,  as  played  or  observed,  during  the  CEP.  Closed-ended  information  components 
included  in  the  questionnaire  are  listed  by  METT-T  factor  in  Table  2.  For  each  component,  the 
questionnaire  asked  the  participants’  to  indicate  whether  that  type  of  information  was  “Required” 
or  “Not  Required.”  For  each  METT-T  factor,  an  open-ended  section  followed  that  asked 
participants  to  identify  any  “Other”  component  types  of  information  required.  For  each 
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component  rated  “Required,”  participants  were  then  asked  to  indicate  whether  that  requirement 
was  “Not  Met”  “Met”  or  “Exceeded.”  For  each  component  rated  “Required”  and  any  “Other” 
components  identified,  participants  were  asked  to  provide  their  recommendations  on  how  that 
requirement  might  be  better  met. 

Table  2 

Information  Components  Used  to  Assess  Battle  Command  Information  Requirements  by 
METT-T  Factor 


Mission 

Enemy 

Terrain 

Troops 

Time 

Warning  Order 
Operations  Order 
Operations  Overlay 
Commander’s  Intent 
Course  of  Action 
Fragmentary  Order 

Location 

Composition 

Disposition 

Ability  to  See 

Ability  to  Move 

Ability  to  Shoot 

Ability  to  Communicate 
Ability  to  Sustain 

Observation 

Cover 

Concealment 

Obstacles 

Key  Terrain 
Approach  Avenues 
Weather 

Location 

Organization 

Ability  to  See 

Ability  to  Move 

Ability  to  Shoot 

Ability  to  Communicate 
Ability  to  Sustain 

Plan 

War  Game 
Prepare 
Rehearse 
Execute 

Sync  Execute 

Although  the  research  issue  addressed  the  information  requirements  of  the  commander, 
the  questionnaire  and  hot  wash  methods  assessed  the  information  requirements  for  each  of  the 
primary  duty  positions  occupied  by  the  test  players.  The  IR  Questionnaire  explicitly  asked  each 
participant  to  assess  how  well  the  information  systems  met  the  information  requirements  of 
“your  duty  position”  during  this  CEP.  The  focus  on  all  test  players  versus  only  the  commander 
(a)  expanded  the  test  players’  sample,  (b)  avoided  confidentiality  issues,  and  (c)  provided  results 
from  key  unit  members  who  were  expected  to  provide  much  of  the  information  required  by  their 
commander. 

Analyses 

Empirical  analyses  were  limited  to  descriptive  statistics,  given  the  preliminary  nature  of 
the  data.  For  this  report,  mean  values  were  computed  for  each  METT-T  component  included  and 
rated  on  the  IR  Questionnaire.  For  each  component,  two  mean  values  were  computed:  one  on 
information  requirement  identified,  and  the  second  on  information  requirements  met.  These 
means  were  computed  across  all  participants  (test  players  and  observers),  and  across  the  three 
test  missions  conducted  during  the  CEP.  Missing  data  was  excluded  for  each  component,  so 
sample  size  varied  below  N  =  90  (30  participants  x  3  missions).  Participants  comments  from  the 
IR  Questionnaire  and  hot  washes  were  subjectively  analyzed  to  identify  unique  statements  and 
provide  approximate  indications  on  the  frequency  of  recurrent  statements.  More  detailed 
analyses  of  the  information  requirements  data  were  performed  for  the  System  Evaluation  Report 
and  Battle  Lab  Evaluation  Final  Report  (Elliott  et  al.,  1998;  USAARMC,  1998).  For  these 
reports,  separate  frequency  distributions  and  mean  values  for  each  IR  component  were  computed 
on  test  player  and  observer  ratings. 
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Method  for  Assessing  the  MDMP 


The  second  research  issue  for  this  CEP  asked  what  are  the  changes  to  the  decision 
making  process  as  a  function  of  the  future  battle  commander’s  display  capabilities.  Two 
methods  were  used  to  address  this  issue.  First,  an  MDMP  Questionnaire  was  developed  to  assess 
changes  in  the  MDMP  and  decision  making  effectiveness  and  efficiency.  This  questionnaire  was 
administered  to  all  participants;  test  players  (except  UAV  operators)  and  observers,  at  the  end  of 
each  test  mission.  Although  the  sample  size  per  item  varied  because  of  missing  data,  12 
participants  and  around  12  observers  completed  the  questionnaire  for  each  of  the  three  missions 
for  an  overall  N  of  approximately  72.  Second,  test  player  comments  made  during  hot  wash 
discussions  were  recorded. 

Instruments 


The  MDMP  Questionnaire  is  provided  in  Appendix  B.  It  includes  two  parts,  referred  to 
as  MDMP  Step  Assessment  and  MDMP  Efficiency  and  Effectiveness  Assessment. 

MDMP  Step  Assessment.  In  the  first  part,  each  of  the  seven  major  steps  and  underlying 
sub-steps  of  the  current  MDMP  were  listed.  Table  3  contains  a  list  of  the  seven  major  MDMP 
steps  and  their  sub-steps.  The  steps  were  based  on  the  field  manual  (FM)  Staff  Organization  and 
Operations  (FM  101-5),  as  adapted  for  research  and  development  purposes  (Graves,  Campbell, 
Deter,  &  Quinkert,  1997).  For  each  step  and  sub-step,  participants  were  asked  to  indicate 
whether  the  step  or  sub-step  was  “Unchanged,”  “Changed,”  “Combined”  or  “Eliminated.” 
Definitions  of  these  responses,  and  a  sample  item,  as  it  appears  in  the  questionnaire,  are  given  in 
Table  4.  If  a  step  was  rated  as  “Unchanged,”  all  its  sub-steps  were  automatically  considered  as 
“Unchanged.”  If  a  step  was  rated  as  “Eliminated,”  all  associated  sub-steps  were  automatically 
considered  “Eliminated.”  Responses  were  re-coded,  when  necessary,  to  conform  to  these 
instructions.  After  rating  a  step  and  its  sub-steps,  participants  were  asked  to  provide  written 
comments  for  any  step  or  sub-step  rated  as  “Changed”  or  “Combined.”  Comments  clarified  how 
the  step  or  sub-step  had  been  “Changed”  or  “Combined.” 

MDMP  Efficiency  and  Effectiveness  Assessment.  In  the  second  part  of  the 
questionnaire,  participants  were  asked  to  rate  both  the  effectiveness  and  efficiency  of  the  seven 
major  MDMP  steps,  as  performed  in  the  CEP,  on  a  3-point  scale  (1  =  “Inefficient”  or 
“Ineffective”;  2  =  “Moderately  Efficient”  or  “Moderately  Effective”;  3  =  “Very  Efficient”  or 
“Very  Effective”). 
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Table  3 


Steps  and  Sub-steps  for  the  Military  Decision  Making  Process 


Military  Decision  Making  Process  (MDMP) 

Mission  Analysis 

Prepare  initial  Intelligence  Preparation  of  the  Battlefield 

Analyze  higher  mission 

Analyze  Brigade  (Bde)  order 

Assess  risk 

Briefing  and  approval 

Course  of  Action  Development 

Analyze  Commander’s  (Cdr’s)  guidance 

Analyze  relative  combat  power 

Generate  conceptual  possibilities 

Develop  scheme  of  maneuver 

Incorporate  all  battlelield  operating  systems 

Wargaming 

Follow  wargaming  method 

Employ  all  friendly  and  enemy  forces/resources 

Identify  critical  events  and  decision  points 

Record  results 

Apply  action/reaction/counteraction 

Assess  results 

Course  of  Action  Comparison 

Select  comparison  method  H 

Determine  decision  criteria  and  assign  weighting  values  to  criteria 

Record  results  (COA  decision  matrix) 

Orders  Preparation 

Incorporate  Cdr’s  guidance  | 

Identify  who,  what,  where,  when  &  why 

Incorporate  battlefield  operating  systems  plans 

Review  &  approve 

Rehearsal 

Define  the  standard 

Orient  participants  to  terrain 

Verbalize  concept  of  the  operation  1 

Select  key  events  to  rehearse 

Describe  enemy  CO  As 

Focus  on  purpose  of  rehearsal  (Decision  Support  Template,  branches  &  Sequels, 
synchronization) 

Mission  Execution  and  Transition 

(No  sub-steps  for  this  step) 
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Table  4 


Questionnaire  Response  Definitions  and  Example  Item 


UNCHANGED  -  The  entire  step  was  performed  the  same  way  in  the  restructured  TOC  as  in  a 
conventional  TOC.  If  you  check  “unchanged”  for  the  step,  do  not  check  any  blocks  for  the  sub-step 
under  that  step.  That  is,  if  the  entire  step  is  unchanged,  all  sub-steps  must  also  be  unchanged. 


CHANGED  -  The  step  or  sub-step  was  still  performed,  but  in  a  different  way  in  the  restructured  TOC 
than  in  a  conventional  TOC.  Under  the  comments  section,  please  describe  how  the  step  or  sub-step  was 
performed  differently.  Individual  sub-steps  under  the  step  may  be  unchanged,  changed,  combined  or 
eliminated. 


COMBINED  -  The  step  or  sub-step  was  performed  as  a  part  of  another  step  or  sub-step  in  the  MDMP  in 
the  restructured  TOC.  Under  the  comments  section,  indicate  which  other  step  or  sub-step  was  combined 
with  it.  Again,  sub-steps  under  the  step  may  be  unchanged,  changed,  combined  or  eliminated. 


ELIMINATED  -  The  entire  step  no  longer  needs  to  be  performed  at  all  in  the  restructured  TOC.  If  you 
check  “eliminated”  for  the  step,  do  not  check  any  blocks  for  the  sub-step  under  that  step.  That  is,  if  the 
entire  step  is  eliminated,  all  sub-steps  must  also  be  eliminated. 


UNCHANGED 

CHANGED 

COMBINED 

ELIMINATED 

MISSION  ANALYSIS 

-  Prepare  initial  Intelligence 

Preparation  of  the 

Battlefield 

-  Analyze  higher  mission 

-  Analyze  Bde  order 

-  Assess  risk 

-  Briefing  and  approval 

Comments:  (required  if  changed  or  combined  are  checked)  Also,  please  include  the  addition  of 
any  related  new  steps  or  sub-steps. 


Analyses 

Empirical  analyses  were  limited  to  descriptive  statistics.  Frequency  distributions  were 
computed  for  responses  to  each  MDMP  step  and  sub-step,  by  test  players  and  observers.  Means 
were  calculated  for  effectiveness  and  efficiency  ratings  for  each  step,  by  test  players  and 
observers.  A  subjective  content  analysis  was  performed  on  the  open-ended  comments 
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concerning  changes  in  the  MDMP.  These  are  combined  comments  from  both  test  players  and 
observers.  In  addition,  comments  concerning  the  MDMP  during  hot  washes  were  recorded  and 
reviewed.  A  synopsis  of  these  hot  wash  comments  is  also  presented. 

Preliminary  Results 

This  section  provides  CEP  results  on  battle  command  information  requirements  and  the 
MDMP.  The  results  provided  are  based  on  test  player  and  observer  responses  to  the  IR  and 
MDMP  Questionnaires  and  test  players’  hot  wash  comments. 

Information  Requirements 

A  summary  presentation  of  the  findings  on  METT-T  information  requirements  identified 
is  provided  in  Figure  3.  This  figure  provides  mean  ratings  for  each  component  specified  in  the 
IR  Questionnaire  based  on  a  two-level  scale  in  which  “Not  Required”  =  0,  and  “Required”  =  1 . 

A  summary  presentation  of  the  findings  on  METT-T  information  requirements  met  is  provided 
in  Figure  4.  This  figure  provides  mean  ratings  for  each  information  component  specified  in  the 
IR  Questionnaire  based  on  a  three-level  scale  in  which  “Not  Met”  =  1,  “Met”  =  2,  and 
“Exceeded”  =  3.  A  more  detailed  presentation  of  these  results,  including  frequency  distributions 
for  each  response  category,  are  available  (Elliott  et  al.,  1998). 

Overall,  the  results  indicated  that  nearly  all  of  the  component  types  of  METT-T 
information  included  in  the  IR  Questionnaire  were  regarded  as  required  information,  and  that 
many  of  these  information  components  were  rated  as  met  or  nearly  met  by  the  C4I  systems  used 
in  the  CEP.  Participants  also  identified  “other”  information  components  that  were  required  but 
not  specified  on  the  IR  Questionnaire,  and  participants  provided  many  useful  recommendations 
for  improving  C4I  systems  to  better  meet  their  information  requirements. 

More  detailed  participants’  comments  from  the  IR  Questionnaire  and  hot  wash  comments 
are  considered  in  the  discussion  section.  These  comments  address  “other”  information 
requirements  identified  by  the  participants  but  not  specified  in  the  IR  Questionnaire,  and 
participants’  recommendations  on  how  their  information  requirements  might  be  better  met.  The 
remainder  of  this  section  will  present  results  in  the  manner  obtained  from  the  IR  Questionnaire, 
by  the  factors  of  METT-T. 

Mission 


The  types  of  mission  information  components  specified  in  the  IR  Questionnaire  were 
generally  identified  as  required  information  (see  Figure  3).  The  average  mean  requirement  rating 
across  all  mission  components  was  .91.  The  mean  component  requirement  ratings  ranged  from 
.87  for  fragmentary  order  to  .99  for  operations  overlay.  Participants  also  identified  other  types  of 
required  mission  information,  not  included  in  the  IR  Questionnaire.  Numerous  participants 
noted  that  typical  missions  routinely  entail  additional  and/or  more  detailed  information 
requirements  than  the  CEP  test  missions.  They  stressed  that  information  requirements  in  more 
typical  combined  arms  operations  include  the  planning  and  coordination  of  additional  assets 
from  brigade  such  as  artillery  and  close  air  support,  as  well  as  supporting  assets  such  as  combat 
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Requirement  Ratings:  0  =  Not  Required;  1  =  Required. 

Figure  3.  Information  requirements  identified  by  METT-T  factor  and  component. 

support,  and  combat  service  support.  Similarly,  participants  repeatedly  noted  that  their 
information  requirements  are  generally  intensified  when  there  is  more  detailed  planning  and 
coordination  of  lower  echelon  units  at  company  and  platoon  level. 

Participants  indicated  that  their  mission  information  requirements  were  never  fully  met 
(see  Figure  4).  The  average  mean  met  rating  across  all  mission  components  was  1.91.  The  mean 
component  met  ratings  ranged  from  1.82  for  operations  order  to  1.98  for  operations  overlay.  In 
sum,  participants’  ratings  indicated  their  mission  information  requirements,  as  tested,  only 
approached  being  met. 


The  types  of  enemy  related  information  components  specified  in  the  IR  Questionnaire 
received  very  mixed  ratings  as  required  information  (see  Figure  3).  The  average  mean 
requirement  rating  across  all  enemy  components  was  .79.  The  mean  component  requirement 
ratings,  however,  ranged  from  .47  for  ability  to  communicate  to  .98  on  location.  In  particular, 
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Requirement  Ratings:  1  =  Not  Met;  2  =  Met;  3  =  Exceeded. 

Figure  4.  Information  requirements  met  by  METT-T  factor  and  component. 

participants  did  not  strongly  endorse  a  requirement  for  information  on  ability  to  communicate  or 
ability  to  sustain.  Participants  also  identified  other  types  of  required  enemy  information.  Several 
participants  noted  the  requirement  for  information  on  assessing  battle  damage  to  the  enemy. 

More  specific  recommendations  suggested  this  assessment  might  distinguish  catastrophic  versus 
mobility  kills.  Participants  also  noted  the  need  for  information  on  enemy  course  of  action.  For 
example,  one  participant  commented  that  “what  do  we  expect  the  enemy  to  do  is  not  addressed.” 
Numerous  participants  reported  that  information  on  enemy  dismounts  was  neither  met  nor 
attempted  by  their  C4I  systems. 

Participants  provided  mixed  ratings  on  how  well  their  enemy  information  requirements 
were  met  (see  Figure  4).  The  average  mean  met  rating  across  all  enemy  components  was  1.90. 
The  mean  component  met  ratings,  however,  ranged  from  1.60  for  ability  to  sustain  to  2.14  for 
location.  Information  on  enemy  ability  to  shoot,  communicate  and  sustain  received  particularly 
low  ratings.  In  sum,  participants’  ratings  indicated  their  enemy  information  requirements,  as 
tested,  were  met  only  for  location,  composition  and  disposition.  Recall  that  many  participants 
did  not  identify  ability  to  communicate  or  ability  to  sustain  as  required  information.  Participants 
who  did,  however,  strongly  indicated  that  these  requirements  were  not  met. 


16 


Terrain 


The  types  of  terrain  related  information  components  specified  in  the  IR  Questionnaire 
were  generally  identified  as  required  information  (see  Figure  3).  The  average  mean  requirement 
rating  across  all  terrain  components  was  .88.  The  mean  component  requirement  ratings  ranged 
from  .76  for  weather  to  .97  for  observation.  Participants  also  identified  other  types  of  required 
terrain  information.  Other  terrain  information  requirements  cited  by  participants  included 
trafficability  information,  estimated  rate  of  march,  and  obstacles. 

With  the  exception  of  Observation  information,  participants’  ratings  indicated  that  their 
terrain  information  requirements  were  not  met  (see  Figure  4).  The  average  mean  met  rating 
across  all  terrain  components  was  1.80.  The  mean  component  met  ratings  ranged  from  1.54  for 
weather  information  to  2. 16  for  observation  information.  Overall,  participants’  ratings  of  how 
well  terrain  information  requirements  were  met  were  generally  lower  than  those  of  other 
categories.  Participants  stressed  the  need  for  more  accurate  terrain  data  bases  with  more  variable 
resolution,  for  information  on  obstacles  and  smoke,  and  for  the  ability  to  determine  rates  of 
march  and  mobility  corridors  for  both  enemy  and  friendly  units. 

Troops 


The  types  of  troop  related  information  components  specified  in  the  IR  Questionnaire 
were  generally  identified  as  required  information  (see  Figure  3).  The  average  mean  requirement 
rating  across  all  troop  components  was  .88.  The  mean  component  requirement  ratings  ranged 
from  .80  for  ability  to  sustain  to  .96  for  location.  Participants  also  identified  other  types  of 
required  troop  information.  Numerous  participants  noted  that  information  on  their  unit’s  ability 
to  sustain  operations  was  not  adequately  provided.  In  addition,  they  stressed  the  scenarios  did 
not  focus  on  sustainment  requirements.  More  specific  recommendations  noted  the  need  for 
information  on  close  air  support  and  battle  damage  effects  on  friendly  weapon  systems.  For 
example,  one  participant  stated  “the  commander  would  have  better  situation  awareness  if  the 
crews  could  input  their  damage.”  While  text-based  sustainment  information  on  friendly  weapon 
systems  was  available  via  automated  situation  reports,  at  least  one  participant  commented  that  a 
more  graphic  format  such  as  “gumball”  charts  might  be  provided. 

Participants  indicated  that  their  troop  information  requirements  were  very  close  to  being 
met  (see  Figure  4).  The  average  mean  met  rating  across  all  troop  components  was  1.99.  The 
mean  component  met  ratings  ranged  from  1.62  for  ability  to  sustain  to  2.23  for  location. 
Excepting  information  on  their  unit’s  ability  to  sustain,  participants  indicated  their  CEP 
information  systems  met  or  nearly  met  most  component  types  of  troop  information. 

Time 


The  types  of  time  related  information  components  specified  in  the  IR  Questionnaire  were 
generally  identified  as  required  information  (see  Figure  3).  The  average  mean  requirement  rating 
across  all  time  components  was  .86.  The  mean  component  requirement  ratings  ranged  from  .77 
for  wargame  to  .94  for  prepare  and  execute.  Participants  also  identified  other  types  of  required 
time  information.  Numerous  participants  noted  that  the  requirement  to  synchronize  external 
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resources  was  neither  met  nor  addressed.  For  example,  one  participant  stated  that  close  air 
support  coordination  in  support  of  the  task  force  was  poor.  Numerous  participants  also  stressed 
that  sufficient  time  was  not  provided  for  planning  and  preparation  at  company  and  platoon 
levels.  Participants  also  stated  that  future  exercises  should  provide  a  better  digital  brigade 
operations  order  to  aid  task  force  planning  and  preparation. 

With  the  exception  of  execute,  participants  indicated  that  their  time  information 
requirements  were  never  fully  met  (see  Figure  4).  The  average  mean  met  rating  across  all  time 
components  was  1.92.  The  mean  component  met  ratings  ranged  from  1.78  for  synch  execute  to 
2.06  for  execute.  Participants’  ratings  clearly  indicated  their  time  information  requirements,  as 
tested,  only  approached  being  met. 


MDMP  Results 


Results  are  presented  first  by  MDMP  steps.  Then  rating  results  on  decision  making 
effectiveness  and  efficiency  are  presented,  and  followed  by  hot  wash  results.  Participants’ 
questionnaire  comments  are  integrated  into  the  questionnaire  results  on  MDMP  steps.  Comment 
categories,  and  the  frequencies  with  which  each  were  cited,  are  provided  in  Appendix  C. 

MDMP  Step  Results 

Mission  Analysis.  A  majority  of  all  participants,  (test  players  and  observers)  rated  the 
step  Mission  Analysis  as  Changed”  (see  Figure  5).  However,  the  response  to  sub-steps  was 
somewhat  different.  First,  considering  test  players’  responses  for  Mission  Analysis,  there  were 
two  sub-steps  (Prepare  Initial  Intelligence  Preparation  of  the  Battlefield  [EPB]  and  Briefing  and 
Approval)  where  a  majority  of  responses  were  “Changed.”  However,  for  the  three  other  sub¬ 
steps  (Analyze  Higher  Mission,  Analyze  Brigade  Order  and  Analyze  Risk)  a  majority  of 
responses  were  “Unchanged.”  For  observers,  half  or  more  of  the  responses  were  “Unchanged” 
for  all  sub-steps  except  for  Prepare  Initial  IPB  where  the  majority  of  responses  were  “Changed.” 
A  subjective  content  analysis  of  the  written  comments  was  conducted  to  clarify  this  pattern. 
Many  of  the  comments  indicated  that  the  IPB  will  be  changed  significantly  by  the  technology 
used  in  the  CEP.  Participants  noted  that  there  was  far  more  information  on  the  actual  enemy 
situation  available  during  the  planning  phase  than  there  is  in  a  conventional  TOC.  Therefore 
participants  stated  that  the  process  was  considerably  streamlined.  Participants  also  noted  that 
there  was  no  formal  mission  analysis  brief,  thus  accounting  for  the  “Changed”  rating  for  the 
Briefing  and  Approval  sub-step  by  test  players.  However,  the  MDMP  sub-steps  of  Analyze 
Higher  Mission,  Analyze  Brigade  Order  and  Assess  Risk  were  seen  as  “Unchanged.”  In 
particular,  participants  ratings  indicated  that  their  analysis  of  the  brigade  mission  and  order  were 
unchanged. 

Course  of  Action  Development.  Only  test  players  rated  the  Course  of  Action 
Development  step  itself  as  “Changed”  (see  Figure  6),  but  both  test  players  and  observers  rated  all 
sub-steps  as  Unchanged.”  Although  the  majority  of  test  players’  responses  indicated  that  the 
Course  of  Action  Development  step  was  “Changed,”  the  modal  response  for  all  the  sub-steps 
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Figure  6.  Responses  to  course  of  action  development. 
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COA  Dev  Course  of  Analysis  Development 
Guidance  Analyze  Commander’s  Guidance 
Power  Analyze  Relative  Combat  Power 
Possibilities  Generate  Conceptual  Possibilities 
Maneuver  Develop  Scheme  of  Maneuver 
BOS  Incorporate  All  BOS 


was  “Unchanged.”  The  pattern  for  observers  was  the  same,  but  half  the  observers  rated  the  step 
itself  (i.e.,  Course  of  Action  Development)  as  “Unchanged.”  Comments  emphasized  that  while 
the  tools  provided  a  capability  to  develop  effective  courses  of  action,  several  battlefield  operating 
systems  were  not  well  integrated. 


Wargaming.  Test  players  were  somewhat  more  likely  to  rate  Wargaming  as  “Changed” 
in  the  CEP  than  were  observers  (see  Figure  7).  A  solid  majority  of  test  players’  responses 
indicated  that  Wargaming  was  “Changed.”  However,  of  the  six  sub-steps,  the  modal  response 
was  “Unchanged”  except  for  the  sub-steps  of  Follow  Wargaming  Method  and  Record  Results. 
For  these  two  sub-steps  the  modal  response  was  “Changed.”  For  observers  the  modal  response 
was  “Unchanged”  for  the  Wargaming  step  and  all  sub-steps  except  for  Record  Results.  Here,  the 
majority  of  responses  was  “Eliminated.”  Many  comments  noted  that  the  wargaming  method 
followed  in  the  CEP  was  much  abridged  or  less  formal,  which  may  account  for  the  test  player 
ratings  of  the  wargaming  method  as  “Changed.”  Other  frequent  comments  referred  to  use  of  the 
Whiteboard  and  involvement  of  company  commanders. 
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Figure  7.  Responses  to  wargaming. 
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Course  of  Action  Comparison.  Participants  tended  to  see  the  step  Course  of  Action 
Comparison  as  “Changed”  (see  Figure  8),  but  the  sub-steps  as  “Unchanged.”  The  modal 
response  of  all  participants  for  the  step  of  Course  of  Action  Comparison  was  “Changed.”  Again, 
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however,  the  modal  response  for  all  participants  was  “Changed”  for  only  one  of  the  three  sub¬ 
steps  (Select  Comparison  Method).  The  modal  response  for  both  groups  for  the  other  two  steps 
(Determine  Decision  Criteria  and  Weights  and  Record  Results)  was  “Eliminated.”  The 
comments  reflect  these  ratings.  Numerous  comments  from  participants  stated  that  no  formal 
process  was  used:  no  decision  matrix,  no  formal  criteria,  no  weighting  criteria  recorded,  no 
decision  brief.  Several  of  the  comments  implied  that  these  methods  were  unnecessary,  as  the 
solution  was  transparent. 
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Figure  8.  Responses  to  course  of  action  comparison 

Orders  Preparation.  Participants  tended  to  view  the  step  Orders  Preparation  as  “Unchanged”  (see 
Figure  9).  For  Orders  Preparation  a  majority  of  both  test  players’  and  observer  ratings  for  both 
the  step  and,  thereby,  all  sub-steps  was  “Unchanged.”  There  were  fewer  comments  concerning 
this  step  than  any  other  steps.  Comments  described  the  digital  nature  of  the  battalion  order  as 
developing  more  quickly  than  a  conventional  order  and  as  event  driven. 
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Figure  9.  Responses  to  orders  preparation. 
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Rehearsal.  Participants  mostly  saw  the  step  Rehearsal  as  “Unchanged”  (see  Figure  10). 

A  majority  of  test  players’  responses  for  the  step  itself  were  “Changed.”  For  observers,  the 
preponderance  of  responses  for  the  step  were  evenly  divided  between  “Unchanged”  and 
“Changed.”  However,  for  both  test  players  and  observers,  a  majority  of  responses  for  all  sub¬ 
steps  was  “Unchanged.”  The  fact  that  test  players  rated  the  step  as  “Changed”  while  rating  all 
sub-steps  as  “Unchanged”  may  indicate  a  problem  with  the  sensitivity  of  the  questionnaire.  A 
large  number  of  comments  stated  that  the  rehearsal  was  conducted  with  company  commanders 
via  Whiteboard  and  radio.  Other  comments  indicated  that  there  was  no  formal  decision  support 
template.  These  comments  reflect  that  the  task  remains  the  same,  while  techniques  and 
procedures  differ. 

Mission  Execution  and  Transition.  Test  players  and  observers  appeared  to  differ 
concerning  whether  Mission  Execution  was  “Changed”  (see  Figure  1 1).  For  test  players,  the 
majority  response  for  the  Mission  Execution  step  of  the  MDMP  was  “Changed.”  However,  for 
observers,  the  majority  response  was  “Unchanged.”  Many  comments  referred  to  how  indirect 
fires  were  handled  in  the  CEP,  indicating  that  the  artillery  BOS  process  was  quite  different  in  the 
digital  environment.  Another  comment  was  that  the  commander’s  battlefield  visualization  was 
much  greater  during  execution  than  in  a  conventional  TOC.  Other  comments  stressed  the  fact 
that  less  “up  front”  planning  was  necessary,  since  the  staff  could  now  more  easily  detect  and 
react  to  changes  in  the  situation. 
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Figure  10.  Responses  to  rehearsal. 
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Figure  11.  Responses  to  mission  execution  and  transition. 
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Effectiveness  and  Efficiency  Ratings 


The  mean  observer  ratings  on  effectiveness  and  efficiency  appear  uniformly  lower  than 
those  of  test  players.  For  test  players,  ratings  were  slightly  above  “Moderately  Efficient”  or 
“Moderately  Effective”  (an  example  of  the  scales  is  given  at  Figure  12).  For  observers,  ratings 
were  at  or  slightly  below  “Moderately  Efficient”  or  “Moderately  Effective.”  Among  test  player 
ratings,  relatively  lower  ratings  were  assigned  for  wargaming  efficiency,  and  course  of  action 
comparison  effectiveness  and  efficiency.  Similarly,  among  observer  ratings,  relatively  lower 
ratings  were  assigned  for  wargaming  effectiveness  and  efficiency  and  course  of  action 
comparison  effectiveness  and  efficiency. 
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Figure  12.  Efficiency  and  effectiveness  rating. 
Hot  Wash  Results 


In  the  final  hot  wash,  a  version  of  the  MDMP  that  reflects  the  digital  capability  of  the  C4I 
systems  was  reviewed.  This  version  was  based  on  test  players’  experiences  and  inputs  in  the 
CEP  (see  Figure  13).  The  subsequent  description  of  this  digital  MDMP  is  based  on  the  hot  wash 
comments  of  observers  and  test  players,  and  not  responses  to  any  instrument  designed  by  ARI. 
Initially,  the  brigade  digital  order  was  transmitted  to  the  plans  node,  which  in  turn  emailed  the 
OPORD  to  the  other  three  nodes.  The  commander  then  provided  initial  guidance  to  the  Plans 
node  for  COA  development  and  to  the  operations  node  for  reconnaissance  and  surveillance 
planning.  The  plans  node  then  conducted  mission  analysis  followed  by  COA  development, 
analysis  and  comparison  (recommendation).  Meanwhile  the  operations  node  developed  and 
executed  the  reconnaissance  and  surveillance  plan.  Generally,  only  one  COA,  with  embedded 
contingencies,  was  generated.  The  commander  then  modified  the  COA  and  wargamed  it  with 
the  company  commanders  using  the  Whiteboard  and  radio  communication.  The  Plans  node 
eavesdropped  on  the  wargaming  session,  and  quickly  modified  the  plan  based  on  results  of  the 
session.  The  Plans  node  then  transmitted  the  battalion  OPORD  via  the  Whiteboard  to  the 
commander,  who  then  modified  and  issued  the  OPORD.  The  task  force  commander  then 
rehearsed  the  order  with  company  commanders,  again  via  Whiteboard  and  radio. 

Test  players  noted  that  “rehearsals”  were  more  like  a  mission  brief,  perhaps  because 
technical  shortcomings  limited  test  player  interaction.  However,  the  power  of  the  tools  was 
illustrated  during  the  rehearsal  of  the  final  (Attack)  test  mission,  where  the  PVD  revealed  a  major 
change  in  the  way  enemy  troops  were  deployed  based  a  recent  update  via  the  sensors.  The 
enemy  operations  officer  in  the  commander’s  node  called  the  change  to  the  commander’s 
attention,  who  then  altered  the  plan  immediately.  The  new  plan  was  then  executed. 
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The  discrepancy  of  activity  among  nodes  is  readily  apparent  from  Figure  13.  Across  all 
test  missions,  test  players  noted  that  the  commander’s  node  and  operations  node  were  most 
active  in  the  mission  execution  phase.  About  half  way  through  execution  of  the  first  two  test 
missions,  a  new  brigade  mission  (OPORD)  was  emailed  to  the  plans  node,  starting  the  process 
over. 


Discussion 

This  section  discusses  the  CEP  results  on  battle  command  information  requirements  and 
the  MDMP.  First,  this  discussion  provides  some  probable  explanations  for  some  notable  results 
reported  on  identifying  and  meeting  information  requirements.  Then  recommendations  on  how 
information  requirements  might  be  better  met  are  considered  in  more  detail.  Discussion  of  the 
MDMP  results  summarizes  and  interprets  results  from  the  MDMP  Questionnaire  and  test 
players’  hot  wash  comments. 


Information  Requirements 


Assessment 


Overall,  the  results  indicated  that  most  of  the  component  types  of  METT-T  information 
included  in  the  Information  Requirements  Questionnaire  were  regarded  as  required  information. 
At  the  information  component  level,  mean  participants’  ratings  on  information  required 
(“Required”  =  1.00)  ranged  from  .47  for  threat’s  Ability  to  Communicate,  to  .99  for 
Operations  Overlay.  Across  components,  at  the  METT-T  factor  level,  enemy  information 
received  the  lowest  required  mean  rating  at  .79,  and  mission  information  received  the  highest  at 
.9 1 .  Notable  exceptions,  from  the  overall  pattern  that  most  information  components  were  rated 
as  required,  include  selected  component  types  of  enemy  information. 

There  are  several  probable  reasons  for  such  exceptions.  Given  the  preliminary  nature  of 
these  results,  no  conclusions  will  be  made  concerning  information  not  required.  Enemy 
information  is  almost  always  priority  information,  as  indicated  by  the  Commander’s  Critical 
Information  Requirements  (CCIR).  Participants’  high  ratings  on  location,  composition,  and 
disposition  components  reflect  the  requirement  for  such  enemy  information.  In  contrast,  lower 
ratings  occurred  on  enemy  components  such  as  ability  to  communicate  and  ability  to  sustain. 
Why?  Such  ratings  may  indicate  that  current  training  does  not  adequately  stress  the  impact  of 
realistic  communications,  including  degraded  communications,  on  friendly  and  enemy 
capabilities.  In  addition,  questionnaire  and  hot  wash  comments  indicated  that  some  participants 
had  no  idea  how  information  on  these  enemy  components  might  be  obtained  and  provided. 
Participants’  perceptions  about  what  information  might  be  feasible,  therefore,  may  have 
tempered  some  requirement  ratings.  An  alternative  explanation  is  that  the  forward-edge  of  the 
battlefield  role  simulated  by  the  test  participants  may  direct  warfighters'  information 
requirements  to  more  immediate  threat  aspects,  such  as  enemy  location  and  firepower  rather  than 
communication  and  sustainment.  Future  research  efforts,  therefore,  might  take  a  more  precise 
approach  to  identifying  and  assessing  information  requirements  by  task  and  condition. 
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Overall,  the  results  indicated  that  test  participants’  information  requirements,  as  tested, 
were  met  or  nearly  met.  At  the  information  component  level,  mean  participants’  ratings  on  how 
well  their  information  requirements  were  met  (“Met”  =  2.00)  ranged  from  1.54  for  weather,  to 
2.33  for  friendly  location.  Across  components,  at  the  METT-T  factor  level,  terrain  information 
received  the  lowest  met  mean  rating  at  1.80,  and  troop  information  received  the  highest  at  1.99. 
Notably,  none  of  the  information  components  rated  during  the  evaluation  approached  the  level  of 
“Exceeded”  (=  3.0).  These  results  suggest  there  is  a  substantial  need  for  C  i  system 
improvements  to  better  meet  information  requirements. 

Finally,  although  the  evaluation’s  lead  issue  addressed  the  commander’s  information 
requirements,  these  results  were  based  on  the  participants’  duty  position,  as  played  or  observed, 
during  the  evaluation.  Future  efforts  might  assess  the  commander’s  information  requirements 
more  directly,  to  include  the  CCER..  For  example,  questionnaires  might  ask  the  commander  to 
identify  his  CCER  and  to  have  him  and  the  other  participants  assess  how  well  these  CCIR  were 
met.  Future  C4I  systems,  however,  may  affect  CCER  nature  and  process.  These  and  related 
doctrine  and  tactics  issues  might  be  addressed  in  future  efforts. 

Recommendations 


Participants’  recommendations  on  how  to  improve  C4I  systems  to  better  identify  and 
meet  battle  command  information  requirements  are  considered  in  this  section.  Table  5  provides 
a  complete  account  of  participant  recommendations  from  the  IR  Questionnaire,  by  METT-T 
information  component.  These  recommendations  provide  a  fairly  specific  set  of  requirements 
for  improving  the  C4I  systems  used  in  the  CEP.  More  generally,  these  recommendations  may 
provide  useful  information  for  improving  legacy  C4I  systems  as  well. 
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Table  5 


Recommendations  on  Information  Requirements 


Mission  Component  Recommendations  on  Information  Requirements 

Warning  Order 

Comments  on  warning  order  information  repeatedly  stressed  that  a  more  compatible 
set  of  C4I  capabilities  and  products  at  brigade  level  are  needed  to  provide  required 
information.  Recommendations  for  improving  these  brigade  level  products 
included  a  standardized  format  with  more  detail  and  the  ability  to  manipulate  the 
detail  provided.  With  respect  to  standardized  format,  for  example,  one  respondent 
stated  "establish  format  and  method  to  ensure  completeness  of  data."  In  particular, 
respondents  noted  the  need  to  provide  more  precise  information  that  would  inform 
sensor  operators,  such  as  unmanned  aerial  vehicle  operators,  with  immediate  areas 
of  concern.  More  specifically,  some  respondents  urged  that  sensor  operators  be 
issued  a  warning  order.  More  generally,  respondents  also  urged  that  received 
warning  orders  should  include  intelligence  information  on  the  unit’s  area  of 
operation.  With  respect  to  manipulation,  for  example,  one  respondent  stated  "make 
it  [a]  matrix,  pull  out  needed  information."  Another  respondent  stated  "provide 
fthe]  computer  the  ability  to  break  down  higher  order." 

Operations 

Order 

Comments  on  operations  order  information  also  stressed  that  a  more  compatible  set 
of  C4I  capabilities  and  products  at  brigade  level  are  needed  to  provide  required 
information.  In  particular,  respondents  urged  that  C4I  systems  provide  the  ability  to 
extract  or  parse  those  portions  of  a  higher  echelon  order  most  pertinent  to  lower 
echelons.  Test  players  and  O/Cs  urged  a  standardized  format  to  better  ensure  that 
the  detail  missing  from  these  products  during  the  CEP  was  routinely  included.  A 
number  of  respondents  also  urged  a  wider  dissemination  of  the  brigade  level 
operations  order.  For  example,  one  respondent  stated  "why  only  [a]  select  few, 
shouldn’t  everybody  in  the  XO’s  TOC  receive?”  Finally,  several  respondents 
seemed  to  question  the  requirement  for  the  more  traditional,  analog  format  of  a  five 
paragraph  operations  order.  Typical  of  such  comments,  one  respondent  stated  "oral 
only  if  necessary."  One  rationale  for  such  comments,  provided  by  respondents,  was 
that  the  digital  operations  overlay  format  that  evolved  during  the  course  of  the  CEP 
provided  much  of  an  order's  information  in  a  more  useful  format. 

Operations 

Overlay 

Comments  on  operations  overlay  information  again  stressed  that  a  more  compatible 
set  of  C4I  capabilities  and  products  at  brigade  level  are  needed  to  provide 
information.  To  facilitate  visualization  of  the  mission,  respondents  also 
recommended  a  standard  library  of  icon  symbology  and  more  than  one  color  to  ! 

represent  overlay  content  and/or  type.  The  most  recurrent,  and  possibly  the  most  ; 

telling,  comments  on  meeting  mission  information  requirements  centered  on  the 
need  for  a  more  integrated  C4I  tool  set  to  fuse  information  sources.  In  particular, 
respondents  stressed  that  ancillary  tools  such  as  the  Whiteboard  must  be  combined 
with  a  dynamic  map  display.  The  recommendation  to  integrate  all  mission  related 
information  into  dynamic  map  overlays  and  displays  was  strongly  reinforced  by 
both  questionnaire  and  hot  wash  comments. 
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Table  5  (continued) 


Commander’s 

Intent 

Comments  on  information  requirements  related  to  commander’s  intent  stressed  that 

C4I  capabilities  should  enable  a  clear  delineation  of  priorities  and  end  state.  For 
example,  several  respondents  requested  a  format  that  included  more  precise 
information  on  the  priorities  for  unmanned  aerial  vehicle  operators.  Others 
requested  more  precise  information  on  the  commander’s  willingness  to  lose  assets. 

A  number  of  respondents  cited  that  formats  should  provide  a  clear  understanding  of 
the  intended  end  state  of  the  mission.  Generally,  respondents’ comments  indicated 
their  approval  of  the  Whiteboard  link  that  allowed  the  commander  to  depict  and 
annotate  a  sketch  of  his  intent  to  all  test  players. 

Course  of 
Action 

Comments  on  requirements  related  to  course  of  action  information  stressed  that  the 
CEP’s  C4I  capabilities  did  not  provide  a  standardized  format  for  situation  templates 
or  the  ability  to  wargame  courses  of  action.  In  particular,  hot  wash  comments 
underscored  the  need  for  information  formats  to  delineate  and  coordinate  the 
sequence  of  actions.  Numerous  respondents  stressed  the  need  for  a  reconnaissance 
and  surveillance  plan  to  coordinate  actions  and  delineate  control.  Similarly, 
company  commanders  repeatedly  reported  they  understood  the  course  of  action  in 
general,  but  did  not  clearly  understand  the  sequence  and  interaction  for  company 
operations.  A  number  of  respondents  requested  more  specific  information  on 
routes.  For  example,  one  respondent  stated  "did  not  use  any  routes  just  an  area." 
Overall,  respondents  suggested  a  more  continuous  linkage  for  Whiteboard  type 
communications  to  clarify  and  adjust  activities  during  the  course  of  operations. 

Fragmentary 

Orders 

Comments  on  information  requirements  related  to  fragmentary  orders  were  limited. 
Some  respondents  suggested  that  fragmentary  orders  were  not  fully  exercised 
during  the  CEP.  Others  stated  that  oral  orders  followed  the  initial  operations  order. 
Again  numerous  respondents  commented  that  the  information  requirements  related 
to  fragmentary  orders  might  be  better  met  by  providing  a  more  continuous  linkage 
for  Whiteboard-type  communications. 

Enemy  Component  Recommendations  on  Information  Requirements 

Location 

Respondent  comments  stressed  that  the  CEP  simulation  did  not  provide  a 
comprehensive  "read"  on  enemy  location  data.  Numerous  respondents  noted  that  a 
more  extensive  feed  from  higher  echelons  might  have  provided  the  80-90%  solution 
on  all  enemy  locations  they  had  expected.  In  addition,  many  respondents 
complained  that  enemy  location  information  was  not  adequately  shared  among  test 
player  stations.  For  example,  they  reported  that  UAV  operator  stations  only 
displayed  enemy  locations  detected  by  the  UAV  display  in  question.  Finally,  both 
questionnaire  and  hot  wash  comments  suggested  that  future  C4I  systems  should 
dvnamicallv  link  and  match  enemy  situation  templates  to  depicted  enemy  locations. 

Composition 

Comments  on  enemy  composition  cited  the  need  for  additional  information  on 
vehicle  and  unit  types.  For  example,  numerous  respondents  requested  automatic 
identifiers  for  enemy  air  defense  and  logistic  assets.  Others  noted  that  manual 
analysis  and/or  confirmation  of  composition  types  would  still  be  needed  and, 
therefore,  future  C4I  systems  should  at  least  provide  operators  a  labeling  capability 
to  indicate  composition. 

Disposition 

Comments  on  enemy  disposition  suggested  that  better  information  on  enemy 
location  and  composition  would  greatly  assist  efforts  to  determine  disposition. 
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Ability  to  See 

Comments  on  enemy’s  ability  to  see  were  of  two  major  types.  Some  respondents’ 
noted  the  need  for  additional  information  such  as  graphic  depiction  of  the  "dead 
space"  associated  with  enemy  vehicles  or  icons  should  be  provided,  as  required  or 
on-call.  Other  respondents  questioned  the  feasibility  of  the  requirement.  For 
example,  one  respondent  said  "don’t  know  how  you  would  get  this  information  from 
current  system."  While  future  C4I  systems  might  provide  various  levels  of 
information  on  the  enemy’s  ability  to  see,  the  system  used  during  the  CEP  provided 
at  least  one  level.  Test  players  routinely  clicked  on  stationary  friendly  vehicle 
locations  to  obtain  line-of-sight  analyses  on  their  ability  to  see  from  that  location. 
Training  might  have  stressed  this  same  capability  could  be  applied  to  stationary 
enemy  icons  to  obtain  information  on  the  enemy’s  ability  to  see  from  a  selected 
location. 

Ability  to  Move 

Comments  on  enemy’s  ability  to  move  stressed  the  need  for  information  on  potential 
mobility  corridors  for  enemy  units,  and  estimates  on  enemy’s  rate  of  march. 
Numerous  respondents  stressed  the  need  for  more  valid  information  on  the  enemy’s 
ability  to  move.  In  part  these  comments  pertain  to  invalid  or  unrealistic  enemy 
movement  that  many  respondents  perceived  during  the  CEP’s  simulation-based 
missions.  On  a  more  general  level,  such  comments  underscore  the  requirement 
stated  by  numerous  respondents  to  ensure  that  the  information  provided  by  C4I 
systems  is  not  misleading  or  inaccurate. 

Ability  to  Shoot 

Comments  on  enemy’s  ability  to  shoot  stressed  the  need  for  information  on  enemy 
weapon  ranges.  In  particular,  several  respondents  requested  that  C4I  systems  should 
provide  terrain-adjusted  enemy  weapon  ranges.  Numerous  respondents  stressed  the 
need  for  more  valid  information  on  the  enemy’s  ability  to  shoot. 

Ability  to 
Communicate 

Comments  on  enemy’s  ability  to  communicate  included  various  recommendations 
for  better  meeting  this  requirement.  Respondents’ requested  information  to  identify 
vehicles  with  multiple  antennas.  Another  respondent  noted  the  need  for  information 
that  tracked  radio  frequency  and  intensity.  Other  respondents  requested  information 
on  estimated  radio  ranges  adjusted  for  terrain,  and  connectivity  between  vehicles 
and  units. 

Ability  to 
Sustain 

Comments  on  enemy’s  ability  to  sustain  raised  many  questions  about  how  such 
information  might  be  obtained  and  provided.  Several  respondents  noted  that  basic 
information  on  enemy  logistics  status  at  the  start  of  the  mission  might  provide  a 
basis  for  tracking  and  updating  enemy  sustainment.  Generally,  respondents  stressed 
that  mission  design  should  entail  combat  service  support  (CSS)  functions.  Other 
respondents  noted  this  requirement  might  surface  with  more  extended  mission 
operations.  For  example,  one  respondent  noted  "the  fight  never  lasted  long  enough 
to  strain  CSS  functions."  Another  respondent  stated  that  he  "never  saw  enemy 
supply  trains." 

Terrain  Component  Recommendations  on  Information  Requirements 

Observation 

Comments  on  terrain-based  observation  information  requirements  stressed  that 
many  of  the  terrain  tools  provided  were  not  very  effective.  Several  respondents 
were  also  concerned  that  the  tools  provided  would  not  work  in  a  non-desert 
environment  with  more  challenging  relief.  And  several  respondents  iterated  the 
need  for  a  better  data  base  to  support  their  requirements  for  observation. 
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Cover 

Comments  on  terrain-based  cover  information  requirements  stressed  many  of  the 
points  noted  on  the  observation  component.  In  addition,  due  to  the  limited  cover 
available  in  the  area  of  operations  used  for  the  CEP,  many  respondents  suggested 
they  were  not  too  concerned  about  such  information  and  did  not  exercise  some  of 
the  terrain  tools  available.  Again,  several  respondents  expressed  the  need  for  cover 
information  in  more  challenging  relief,  but  concerns  about  the  accuracy  of 
information  that  might  be  provided  in  such  areas. 

Concealment 

Comments  on  terrain-based  concealment  information  requirements  stressed  many  of 
the  points  noted  for  observation  and  cover  components.  In  general,  respondents 
urged  that  more  challenging  relief  be  used  in  future  efforts  to  better  assess  how  well 
terrain  related  information  requirements  are  met.  Again,  respondents  stressed  the 
need  for  accurate  terrain  data  bases. 

Obstacles 

Comments  stressed  that  obstacle  information  was  a  critical  requirement  for  many 
military  operations,  including  those  conducted  during  the  CEP.  They  also  stressed 
that  the  C4I  information  systems  used  during  the  CEP  failed  to  provide  or  depict 
most  of  the  information  required  on  manmade  obstacles.  In  addition,  they  urged 
that  future  trial  scenarios  should  ensure  that  obstacle  emplacement  and  deliberate- 
breaching  tasks  are  required  aspects  of  the  operation. 

Key  Terrain 

Comments  on  key  terrain  information  requirements  noted  that  the  C4I  information 
systems  used  during  the  CEP  provided  no  capability  to  explicitly  identify  or  mark 
key  terrain  features.  Numerous  respondents  expressed  a  belief  that  human  analysis 
is  required  to  accurately  identify  what  terrain  aspects  or  features  are  key.  At  a 
minimum,  however,  they  recommended  future  systems  should  provide  a  means  to 
label  or  mark  key  terrain.  Finally,  several  respondents  commented  that  key  terrain 
markings  would  be  useful  for  mission  analysis  and  brief. 

Approach 

Avenues 

Comments  stressed  that  the  CEP  C4I  capabilities  did  not  support  respondents’ 
information  requirements  for  approach  avenues.  At  a  minimum,  these  comments 
urged  future  systems  should  provide  approved  graphic  symbols  for  approach 
avenues  and  the  ability  to  annotate  and  delete  those  graphics  as  required.  In 
addition,  respondents  suggested  that  future  systems  should  be  able  to  analyze  and 
identify  approach  avenues  automatically,  and  that  this  ability  should  adapt  to 
different  unit  sizes. 

Weather 

Comments  on  weather  information  requirements  noted  that  weather  was  not  a  factor 
during  the  operations  simulated  during  the  CEP.  Scenario  design  did  not  stress 
weather  effects  and  only  clear  daylight  conditions  were  simulated.  Several 
respondents  stated  there  should  be  a  requirement  for  a  weather  report  as  part  of  the 
brigade  operations  order.  One  respondent  noted  that  Doppler-type  radar  updates  on 
weather  conditions  might  be  helpful  in  future  operational  settings. 

Troop  Component  Recommendations  on  Information  Requirements 

Location 

Comments  on  improving  troop  location  information  were  quite  limited  which 
seemed  consistent  with  their  relatively  higher  ratings  on  this  information  component. 
For  example,  several  respondents  stated  "maintain"  the  level  of  information 
provided.  Numerous  respondents  urged  a  more  distinctive  color  scheme  for  friendly 
icons  and  symbols  to  ease  tracking  of  friendly  troop  locations.  Several  respondents 
also  noted  that  the  location  of  command  vehicles  should  be  provided  on  their  tactical 
displays. 
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Organization 

Comments  on  improving  troop  organization  information  stressed  the  need  for  labels 
indicating  unit  affiliations.  Several  respondents  indicated  that  unit  organization 
should  be  part  of  the  operations  order.  Recall,  that  under  mission  information 
requirements  numerous  respondents  urged  a  standardized  format  for  digital 
operations  orders.  Comments  on  organization  requirements  suggested  that  such 
information  should  be  part  of  that  standardized  format.  In  addition,  several 
respondents  noted  difficulty  in  identifying  separate  units. 

Ability  to  See 

Comments  on  the  friendly’s  ability  to  see  were  quite  limited.  One  respondent  noted 
that  this  ability  might  be  improved  by  providing  a  higher  magnification  for  terrain 
data  base  manipulation. 

Ability  to 
Move 

Comments  on  the  friendly’s  ability  to  move  stressed  the  need  for  information  to 
improve  test  players’ understanding  of  trafficability.  Several  respondents  again  cited 
the  need  for  more  automated  information  to  identify  avenues  of  approach  and 
mobility  corridors.  One  respondent  requested  a  capability  that  indicated  current 
range  of  a  vehicle,  based  on  fuel  consumed.  Finally,  one  respondent  noted  that  even 
with  more  automated  information  on  the  unit’s  ability  to  move,  "company 
commanders  [still]  have  to  pave  the  way." 

Ability  to 
Shoot 

Comments  on  the  friendly’s  ability  to  shoot  provided  a  variety  of  suggestions  for 
better  meeting  this  information  requirement.  The  most  frequent  request  was  the 
ability  to  directly  observe  firings  by  direct  and  indirect  systems  on  the  tactical 
display,  a  capability  sometimes  called  "engagement  updates."  Respondents  also 
noted  that  their  information  systems  should  indicate  "dead  space"  in  relation  to 
weapon  systems’ capability. 

Ability  to 
Communicate 

Comments  on  the  friendly’s  ability  to  communicate  noted  that  test  players  had  no 
ability  to  analyze  radio  ranges  or  to  assess  their  need  to  retransmit.  Several 
respondents  stressed  the  need  for  more  immediate  communication  function, 
particularly  for  electronic  mail  messages.  In  general,  respondents  noted  that  the 
simulation,  as  conducted,  resulted  in  "broken"  communications  and  that  future 
efforts  should  improve  on  this  problem. 

Ability  to 
Sustain 

Comments  on  the  friendly’s  ability  to  sustain  forcefully  stressed  that  combat  service 
support  information  is  a  critical  information  requirement  that  was  not  addressed 
during  the  CEP.  Their  comments  also  emphasized  that  the  inclusion  of  logistics  was 
essential  to  CEP  efforts  to  better  identify  and  meet  the  information  requirements 
related  to  sustaining  friendly  operations. 

Time  Component  Recommendations  on  Information  Requirements 

Plan 

Comments  on  time  to  plan  from  many  respondents  stressed  that  no  time  lines  for  task 
force  operations  were  established  during  the  CEP  missions.  Related  comments  noted 
that  the  CEP’s  C4I  systems  provided  no  capability  to  analyze  time  requirements  or  to 
monitor  progress  in  relation  to  a  time  line.  Several  respondents  stated  there  was 
more  than  adequate  time  to  plan,  but  others  stressed  that  the  lack  of  detail  in  task 
force  planning  underestimated  planning  requirements.  Comments  on  the  need  for 
more  detailed  planning  cited  the  requirement  for  more  extended  combined  arms 
planning  across  battlefield  operating  systems,  more  detailed  information  from  higher 
echelons,  and  more  time  for  lower  echelon  planning  at  company  and  platoon  levels. 
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Wargame 

Comments  on  time  to  wargame  were  quite  mixed.  Some  respondents  stated  there 
was  too  much  time  allowed  for  this,  but  another  stated  that  wargaming  was  "quite 
rushed."  At  least  one  explanation  for  these  apparent  discrepancies  might  be  some 
confusion  about  the  nature  of  wargaming.  As  one  respondent  stated  "the  type  and 
function  of  wargame  were  unclear."  Another  explanation  might  be  duty  level 
differences.  In  particular,  lower  level  company  commanders  commented  on  the  need 
to  maintain  adequate  planning  time.  Both  questionnaire  and  hot  wash  comments 
strongly  stressed  that  future  C4I  systems  should  provide  an  automated  capability  to 
wargame  potential  courses  of  action. 

Prepare 

Comments  on  time  to  prepare  were  as  mixed  as  comments  on  time  to  plan.  Notably, 
they  varied  from  "too  much  time  allowed"  to  "need  more  time."  Again  differences  in 
duty  level  might  explain  some  of  this  variation  with  lower  level  company 
commanders  requesting  more  time.  For  example,  one  respondent  stated  "need  more 
time  before  executing  for  company  commander  to  brief  platoon  leaders."  Similarly, 
another  respondent  urged  that  future  efforts  should  "replicate  preparation  at  platoon 
and  crew  level."  In  addition,  several  respondents  repeated  the  need  for  an  established 
time  line. 

Rehearse 

Comments  on  time  to  rehearse  primarily  stressed  the  need  for  more  time  to  rehearse. 
Again,  company  commanders  stressed  the  need  to  rehearse  with  their  platoon  leaders. 
Several  other  respondents  noted  that  rehearsals  were  conducted  at  Task  Force  level 
only.  In  addition,  questionnaire  and  hot  wash  comments  noted  that  future  C4I 
systems  should  provide  some  automated  capability  for  the  unit  to  rehearse. 

Execute 

Comments  on  time  to  execute  primarily  stressed  that  future  C4I  systems  should 
provide  supporting  automated  capabilities.  Both  questionnaire  and  hot  wash 
comments  identified  an  unmet  requirement  for  a  decision  point  tool.  Respondents 
requested  a  capability  to  input  and  display  decision  points  on  the  tactical  display.  In 
addition,  numerous  respondents  requested  that  displayed  decision  points  should  be 
coupled  with  an  automated  check  and  feedback  before  designated  decision  points 
were  reached. 

Synch 

Execute 

Comments  on  time  to  synch  execute  also  stressed  that  future  C4I  systems  should 
provide  supporting  automated  capabilities.  Both  questionnaire  and  hot  wash 
comments  cited  the  need  for  a  digital  version  of  a  synchronization  matrix. 

Suggestions  on  the  format  for  this  digital  synchronization  matrix  varied  considerably, 
but  numerous  respondents  stressed  the  need  for  a  dynamic  format  with  automated 
updates  during  the  course  of  execution.  Others  suggested  that  a  digital 
synchronization  matrix  should  support  alternate  formats  to  include  both  table  and 
figure  presentations  of  the  same  information. 

Overall,  participants’  comments  provided  many  important  recommendations  to  better 
meet  battle  command  information  requirements.  Table  6  provides  a  summary  of  these 
recommendations,  by  METT-T  factor.  This  table  represents  a  summary  of  the  key 
recommendations  from  Table  5  and  the  IR  Questionnaire,  and  participant  recommendations  from 
the  hot  wash  sessions.  Based  primarily  on  participant  comments,  the  Table  6  items  differentiate 
recommendations  directed  at  C4I  system  improvement  versus  scenario  improvement.  The 
scenario  improvement  recommendations  relate  directly  to  the  scenarios  used  during  the  CEP. 
These  recommendations  should  provide  useful  information  for  design  and  refinement  of 
scenarios  for  future  efforts. 
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A  dominant  theme  across  recommendations  was  the  need  to  more  effectively  integrate 
the  information  required  and  provided.  Such  comments  stressed  that  a  dynamic  map  display 
should  be  the  primary  locus  for  visualizing  the  battlefield.  Participants  urged  that  all  information 
tools  directly  link  to  a  dynamic  map  display.  In  addition,  they  advocated  graphic  formats  for 
required  information,  whenever  possible,  with  graphic  links  to  supporting  text  information  and 
data  bases,  as  required. 

Table  6 

Recommendations  on  Meeting  Battle  Command  Information  Requirements  by  METT-T  Factor 


Factor 

Information  Requirement  Recommendations 

Mission 

Compatible  Information  Systems  and  Products  at  Brigade 

More  Detailed  Information  at  Company  and  Platoon 

Electronic  Aids  for  Wargaming 

Automated  Situation  Template 

All  Information  Tools  Integrated  to  Dynamic  Map  Display 

Enemy 

Current  Enemy  Situation  Linked  to  Enemy  Templates 

More  Detailed  Information  on  Composition 

Battle  Damage  Assessment  Updates 

Terrain-Adjusted  Weapon  Ranges 

More  Realistic  Movement  and  Firing 

Terrain 

More  Accurate  Data  Bases  with  Better  Resolution 

Obstacle  Emplacement  and  Breach  Symbology 

Smoke  Effects  Calculated  and  Depicted 

Estimated  Rate  of  March 

Mobility  Corridors  Analyzed  and  Depicted 

Troops 

Combined  Arms  Operations’ 

Sustainment  Conditions  Required* 

Proactive  Brigade  Assets  for  “Shared  Box”* 

Reconnaissance  &  Surveillance  Plan 

Unmanned  Aerial  Vehicle  Coordination 

Time 

£  ^ 

More  Detailed  Information  from  Brigade 

Time  Line  Monitor  and  Feedback 

Company  and  Platoon  Rehearsals* 

Synchronization  Matrix  Tool  with  Table  and  Figure  Formats 
Decision  Point  Display  and  Checks 

*  Note.  Recommendations  more  related  to  improving  scenarios  than  C4!  systems. 
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Recommendations  for  a  dynamic  map  display  also  reflect  participants’  concerns  about 
information  accuracy  and  currency.  Comments  on  linking  the  current  enemy  situation  to  enemy 
templates,  for  example,  stressed  that  C4I  displays  should  maintain  a  current  depiction  of  the 
enemy  situation.  Participants  urged  that  future  systems  continuously  match  and  assess  detected 
vs.  projected  enemy  data.  Hot  wash  discussion  noted  that,  in  the  future,  threat  operations  may 
not  match  contemporary  doctrinal  “templates.”  However,  they  noted  that  the  intelligence 
preparation  of  the  battlefield  process  would  still  entail  the  projection  of  threat  courses  of  action. 
Participants  recommended  that  C4I  systems  routinely  assess  such  projections  against  current 
enemy  data,  and  graphically  highlight  identified  discrepancies. 

Participants’  comments  on  mission-based  information  also  stressed  that  a  more 
compatible  set  of  C4I  capabilities  and  products  at  brigade  are  needed  to  provide  required 
information.  Recommendations  for  improving  these  brigade-level  products  included  a 
standardized  format  with  more  detail  and  the  ability  to  manipulate  readily  the  detail  provided. 
Participants  emphasized  that  compatible  C4I  systems  should  achieve  a  truly  shared  operational 
area  (“box”)  for  brigade  and  battalion  level  units.  The  battalion  commander  stated  this  shared 
box  should  result  in  proactive  vs.  reactive  synchronization  of  brigade  and  above  assets.  Also,  it 
would  substantially  reduce  the  need  for  information  and  support  requests  between  echelons  and 
allow  the  higher  echelon  to  match  resources  to  priorities.  Participants’  comments  on  system 
improvements  also  included  many  human  factors  aspects  such  as  color  and  symbol  codes,  e-mail 
prompts  and  replies,  more  precise  and  easier  input  devices,  standardized  formats  and  products, 
information  data  bases,  and  job  aids. 

Overall,  the  findings  on  information  requirements  met  provide  an  empirical  basis  for 
directing  improvements  in  future  C4I  systems.  Lower  ratings  indicating  unmet  requirements 
provide  one  basis  for  improvement  priority.  Notably,  these  ratings  pertain  only  to  information 
components  previously  rated  as  “Required”  by  a  participant.  Ratings  that  identified  information 
requirements  provide  another  basis  for  improvement  priority.  A  weighted  coupling  of  both  met 
and  identified  ratings  might  also  be  used  for  the  same  purpose. 

Participants  also  stressed  the  need  for  expanded  operational  evaluations  to  better  identify 
and  assess  C4I-based  information  requirements.  Their  comments  on  terrain  information 
requirements  stressed  that  future  evaluations  should  use  more  challenging  relief  and  varied 
operational  settings.  In  addition,  participants  stated  that  more  informative  assessments  should 
entail  continuous  operations,  combined  arms  operations,  futuristic  scenarios  and  capabilities  for 
friendly  and  threat  forces. 

Predictably,  the  CEP  evaluation  raised  rather  than  resolved  many  important  issues 
related  to  doctrine,  tactics,  techniques  and  procedures.  For  example,  participants  stressed  that 
proponents  of  future  C4I  systems  should  not  expect  such  systems  will  supplant  the  expertise 
required  for  visualizing  the  battlefield.  Soldiers  should  always  be  able  to  assess  the  accuracy  and 
quality  of  information  provided  by  an  automated  information  system.  Exposure  to  anticipated 
C4I  capabilities  triggered  participants’  concerns  about  information  verification  and  human 
decision  making.  Rather  than  concerns  about  too  much  information,  participants  stressed 
information  access.  The  evaluation’s  effort  to  expedite  plan  and  prepare  phases  may  have 
prompted  participants’  concerns  about  information  restrictions,  particularly  the  filtering  of 
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information  through  traditional  communications  channels  or  networks.  In  sum,  the  evaluation’s 
findings  indicate  the  potential  impact  of  future  C4I  systems  on  doctrine,  tactics  and  procedures, 
and  especially  information  requirements,  is  extensive  but  largely  unknown. 

MDMP 


MDMP  Step  Results 

There  were  few  major  changes  reported  by  participants  concerning  the  MDMP.  Overall, 
test  players  appeared  more  likely  to  rate  MDMP  steps  as  “Changed”  than  were  observers. 
However,  both  test  players  and  observers  rated  most  sub-steps  as  “Unchanged.”  No  steps  or  sub¬ 
steps  had  a  modal  response  of  “Combined,”  and  only  two  sub-steps  of  COA  comparison  had  a 
modal  response  of  “Eliminated.”  This,  along  with  the  questionnaire  comments,  suggested 
minimal  changes  in  the  MDMP  during  the  CEP.  Many  questionnaire  comments  stress  that  the 
MDMP  steps  were  performed  in  a  less  formal,  more  streamlined  manner.  Nonetheless,  the  steps 
were  still  performed. 

There  are  cogent  reasons  for  the  lack  of  major  changes  in  the  MDMP  in  this  CEP.  The 
digital  orders  provided  by  brigade  did  not  yet  take  full  advantage  of  the  C4I  systems  capabilities. 
Therefore,  the  initial  steps  of  mission  analysis  and  course  of  action  development  may  have 
proceeded  along  more  traditional  lines  than  if  the  orders  enabled  an  integrated  pictorial 
representation  of  the  mission.  Secondly,  the  test  players  lacked  digital  experience  and  were 
provided  with  no  digital  doctrine.  Thus  it  is  to  be  expected  that  they  would  feel  more 
comfortable  with  familiar  doctrine  and  procedures  rather  than  invent  new  doctrine  and 
procedures. 

The  COA  comparison  step  was  the  one  where  most  change  was  reported.  Since  only  one 
COA  was  developed  for  the  most  part,  a  detailed  comparison  between  COA  was  obviated.  Also, 
the  common  picture  of  friendly  and  enemy  provided  by  the  C4I  systems  tended  to  make  the 
choice  of  COA  fairly  obvious,  as  indicated  by  several  participant  comments. 

Effectiveness  and  Efficiency  Results 

Effectiveness  and  efficiency  findings  paralleled  those  of  MDMP  step  ratings.  The 
effectiveness  and  efficiency  ratings  from  all  participants  centered  around  “Moderately  Effective” 
or  “Moderately  Efficient.”  Ratings  however,  were  especially  low  for  the  steps  of  Wargaming 
and  Course  of  Action  Comparison.  This  makes  sense,  since  there  was  only  one  real  COA 
developed.  Thus  a  formal  comparison  to  other  COAs,  with  an  elaborate  decision  process,  was 
unnecessary. 

Hot  Wash  Results 


Hot  wash  results  again  showed  all  MDMP  steps  being  performed,  albeit  some  were  now 
performed  concurrently  at  different  levels.  For  example,  the  OPORD  was  wargamed  with 
company  commanders  and  simultaneously  modified  by  the  plans  node,  rather  than  lock-step  or 
sequential.  The  main  finding  might  be  that  some  steps  such  as  wargaming,  course  of  action 
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comparison,  and  rehearsal  can  be  performed  in  a  more  streamlined,  faster,  less  formal  manner 
with  the  capabilities  provided  in  the  CEP.  Test  players  reported  that  their  ability  to  react  to 
changes  on  the  battlefield  was  substantially  improved  by  the  tools  provided  in  the  CEP. 

Conclusions 

This  section  provides  conclusions  on  the  results  obtained.  It  also  provides  more  extended 
recommendations  on  how  to  improve  methods  for  assessing  battle  command  information 
requirements  and  military  decision  making.  In  particular,  these  conclusions  attempt  to  leverage 
the  power  of  digital  technologies,  such  as  C4I  systems,  to  improve  assessment  methods. 

More  General  Method  Issues 


Method  improvements  for  assessing  battle  command  information  requirements  and 
military  decision  making  entail  more  general  CEP  evaluation  issues  such  as  test  scenario 
development,  training,  and  test  design.  These  broader  method  issues  are  briefly  described,  but 
addressed  in  detail  elsewhere  (Elliott  et  al.,  1998). 

Scenario  design  is  critical  to  improved  measurement  methods.  Information  requirements 
and  decision  making  are  highly  dependent  upon  task  context  and  the  battlefield  situations  that 
comprise  the  test  scenario  or  mission.  Terrain  factors,  for  example,  impact  information 
requirements  significantly.  This  evaluation  relied  on  relatively  sterile  desert  terrain,  but  such 
terrain  is  not  the  only  type  mounted  forces  may  encounter.  Future  efforts  should  include  other 
potential  conflict  areas  such  as  inhabited  and  forested  locations.  Task  context  alters  information 
requirements  and  decision  making  processes  across  scenarios,  even  within  a  scenario.  The  CCIR 
might  change  repeatedly  during  a  mission  on  a  dynamic,  unpredictable  battlefield.  Also,  rapidly 
changing  scenarios  would  require  more  rapid  decision  making  and  likely  require  a  modified 
decision  making  process.  Accordingly,  participants  urged  that  future  evaluations  employ  robust 
and  representative  mission  settings  for  more  valid  findings. 

Training  is  also  key  to  improved  assessment.  Participants  should  possess  needed  skills: 
fundamental  combat  skills  for  assigned  duty  positions,  basic  C4I  operator  skills,  and  the  ability  to 
integrate  combat  and  computer  skills  in  a  warfighting  environment.  Ideally,  training  would  also 
address  higher-order  decision  making  skills,  requiring  test  players  to  understand  the  possibilities 
of  a  C4I  system  for  mission  accomplishment.  As  indicated  in  the  discussion  of  evaluation 
results,  participants  were  not  fully  aware  of  all  the  informational  capabilities  provided  by  their 
C4I  systems  at  the  individual  or  collective  level  (e.g.,  enemy’s  ability  to  see).  Enabling 
participants  to  use  the  C4I  system  to  obtain  an  integrated  representation  of  the  battlefield  could 
likely  further  streamline  the  decision  making  process.  Providing  improved  training,  urged  by 
participants,  is  needed  to  reduce  measurement  error  and  obtain  a  more  valid  assessment  of  how 
well  information  requirements  are  met  and  the  extent  to  which  decision  making  processes  are 
changed. 

Finally,  test  design  for  the  evaluation  should  directly  map  measurement  methods  to 
evaluation  purpose  and  issues.  Design  challenges  include  the  fact  that  concepts  such  as  battle 
command  are  not  well  defined  from  an  operational  measurement  perspective.  Notably,  the 
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exploratory  nature  of  the  CEP  investigation  was  consistent  with  evaluation  schedule  and 
resources  that  precluded  multiple  and  more  exacting  samples  from  intact  participant  units  with 
conventional  and  C4I  system  collective  expertise. 

Future  test  designs,  however,  should  capitalize  on  lessons  learned  during  this  evaluation 
to  assess  information  requirements  and  decision  making.  Test  designs  should  also  capitalize  on 
prior  C4I  system  evaluations,  particularly  the  Army’s  AWEs.  Virtual  simulation,  for  example, 
affords  many  tools  for  exacting  control  over  complex  test  designs  (Elliott,  Sanders,  &  Quinkert, 
1996).  In  addition,  formative  evaluation  methods  for  improving  a  system  should  be  applied 
before  summative  evaluations  are  conducted  (Lickteig,  1996). 

Information  Requirements 


Results 


The  results  indicated  that  most  of  the  component  types  of  METT-T  information  included 
in  the  Information  Requirements  Questionnaire  were  regarded  as  required  information.  Notable 
exceptions  from  the  overall  pattern  include  selected  component  types  of  enemy  information. 
Given  the  preliminary  nature  of  these  results,  however,  no  conclusions  should  be  made 
concerning  information  not  required.  Future  research  efforts,  therefore,  might  take  a  more 
precise  approach  to  identifying  and  assessing  information  requirements  by  task  and  condition. 
This  approach  is  discussed  in  Lickteig  (1996).  Overall,  the  results  indicated  that  test 
participants’  information  requirements,  as  tested,  were  met  or  nearly  met.  These  results  also 
suggest,  however,  there  is  a  substantial  need  for  C4I  system  improvements  to  better  meet 
information  requirements. 

Method  Improvements 

Approach.  Defining  the  domain  and  content  of  user  information  requirements  is  an 
essential  issue  to  improving  measurement  methods  for  C4I-based  operations.  Recall,  the  lead 
research  issue  for  the  evaluation  specified  a  METT-T  approach  for  assessing  information 
requirements.  However,  other  informational  structures  for  defining  combatants’  information 
domain  might  be  used  such  as  tasks,  functions  or  battlefield  operating  systems.  Strategies  for 
restricting  the  domain  of  interest  might  also  be  used.  One  strategy  for  limiting  the  domain  might 
target  the  information  required  for  selected  processes,  products,  or  outcomes  related  to  unit 
performance.  Another  strategy  might  be  to  address  a  requirement  subset,  such  as  the  CCIR. 

While  future  research  efforts  might  carefully  consider  the  tradeoffs  associated  with 
alternate  informational  domain  structures,  there  are  important  reasons  why  the  METT-T 
approach  for  defining  and  assessing  user  information  requirements  is  useful.  For  instance,  the 
METT-T  organization  of  battlefield  information  is  a  fundamental  heuristic  of  Army  training  and 
doctrine.  Also,  from  an  evaluation  and  training  feedback  perspective,  the  representation  and 
manipulation  of  METT-T  information  on  a  C4I  display  provides  a  tractable  and  meaningful  basis 
for  assessing  information  requirements  (Lickteig  &  Throne,  in  preparation).  The  following 

sections  will  discuss  how  METT-T  supports  the  development  and  use  of  manual  and . 

instrumented  measures  for  assessing  information  requirements  and  use. 
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Manual  Measures.  Only  manual  measures  were  used  during  the  evaluation  to  identify 
information  requirements  and  assess  how  well  those  requirements  were  met.  A  lesson  learned 
from  this  evaluation  is  that  both  manual  and  instrumented  measures  should  be  used  for 
information  requirements  assessment.  Lessons  learned  from  this  evaluation  and  related  research 
efforts  for  method  improvements  are  considered  for  manual  measures,  and  then  for  instrumented 
measures. 

A  general  lesson  reinforced  by  the  evaluation  is  that  methods  for  identifying  and 
assessing  information  requirements  are  best  embedded  in  task  context  (Cooke,  1994).  Methods 
for  identifying  information  requirements  during  task  performance,  without  becoming  disruptive, 
might  be  used  (Cooke,  1994).  Often  called  process  tracing,  such  methods  include  manual 
protocols  of  participant  or  expert  observer  verbalizations  and/or  actions  to  request,  access,  use, 
and  provide  information  as  tasks  are  performed  during  a  mission.  An  alternate  method  is  to  elicit 
the  same  types  of  information  during  a  “simulated”  mission  rather  than  actual  missions  to  reduce 
task  interference. 

Another  lesson  reinforced  by  the  evaluation  is  that  the  design  of  manual  measures  should 
match  method  specificity  and  participants’  experience.  In  the  original  version  of  the  Information 
Requirements  Questionnaire  participants  were  asked  to  assess  their  information  requirements  at 
an  overall  METT-T  factor  level.  One  reason  for  that  approach  was  to  reduce  participants’ 
workload  by  limiting  the  number  of  items  to  be  assessed  (5  factors  vs.  34  components).  As  the 
informational  components  underlying  each  factor  had  not  yet  been  assessed,  however,  this  item 
was  too  general  and  abstract.  Measure  design  for  exploratory  research  should  match  the 
participants’  actual  evaluation  experience.  One  strategy  for  matching  test  design  to  test 
experience  is  the  use  of  more  structured  test  scenarios  (Campbell,  Campbell,  Sanders,  Flynn,  & 
Myers,  1995). 

A  related  concern  is  that  information  requirements  are  a  multidimensional  issue.  A  C4I 
system  may  do  a  poor  job  of  providing  information  requirements  if  it  does  not  provide  the 
information,  if  it  requires  the  user  to  perform  too  much  work  to  extract  the  information,  or  if  it 
does  not  provide  the  information  in  a  timely  manner.  The  IR  Questionnaire  focused  primarily  on 
identifying  what  information  was  required.  In  the  future,  assessment  methods  should  also 
address  the  workload  associated  with  the  information  provided  and  not  provided,  to  include 
information  overload.  And  methods  should  also  address  the  usefulness  of  the  information 
provided,  to  include  accuracy,  clarity,  completeness,  and  timeliness.  For  a  more  comprehensive 
set  of  method  recommendations  for  assessing  C4I  information  requirements  see  Lickteig  and 
Throne  (in  preparation). 

In  addition,  the  category  of  "exceeded"  information  requirements  raises  several  concerns. 
First,  the  category  may  introduce  ambiguity.  Its  selection  might  indicate  a  favorable  response 
such  as  the  C  I  system  surpassed  the  rating  of  "met,"  or  a  negative  response  such  as  the  C4I 
systems  provided  too  much  information.  Second,  the  category  may  be  inappropriate  for 
requirement  specification.  Only  the  Army  goal  of  meeting  information  requirements  is  clearly 
justified.  For  the  CEP  results  reported,  readers  are  reminded  that  frequency  distributions  for 
each  response  category  are  available  (Elliott  et  al.,  1998).  For  future  efforts,  evaluators  might 
avoid  use  of  the  "exceeded"  category. 
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Finally,  manual  measures  should  not  overload  participants.  In  addition  to  the  Information 
Requirements  Questionnaire,  participants  were  asked  to  complete,  sometimes  repeatedly,  manual 
measures  addressing  the  military  decision  making  process,  C4I  tools,  multifunctional  roles, 
training  and  experience  requirements,  workload,  and  information  significance  and  structure 
(Elliott  et  al.,  1998).  Test  designers  should  carefully  weigh  participants’  workload  across  all 
proposed  manual  measures,  and  then  develop  and  impose  a  test  design  that  avoids  overloading 
participants.  One  strategy  for  improving  manual  measurement  methods  and  reducing 
participants’  workload  is  a  reliance  on  complementary  instrumented  measures. 

Instrumented  Measures.  The  CEP  evaluation  initiated  instrumentation  of  the  C4I  systems 
to  provide  automated  measurement  methods.  Instrumented  data  from  the  initial  evaluation  was 
not  analyzed  due  to  technical  and  procedural  limitations.  Future  research  efforts,  however, 
should  capitalize  on  the  instrumentation  investment  made  by  the  MBBL  and  test  bed  facility. 

This  section  will  focus  on  the  potential  contribution  of  C4I  instrumented  measures  for  identifying 
and  assessing  information  requirements. 

A  composite  database  of  the  elements  depicted  on  the  tactical  display  by  source  and  path 
represents  a  measurable  product  achieved  through  collaborative  information  processing  and 
management.  This  database  provides  a  quantifiable  link  between  the  informational  requirements 
of  the  user  and  the  informational  capabilities  of  the  tactical  display.  It  also  provides  a  tractable 
link  for  identifying  and  assessing  the  information  required,  available,  and  used. 

An  indirect,  but  telling,  measure  of  information  utilization  is  visual  access.  Information 
available  in  a  system  is  not  always  visible  to  the  user.  Although  troop  and  enemy  location 
information  was  uniformly  available  to  all  participants  in  the  CEP,  visual  access  to  that 
information  was  determined  by  the  map  area  currently  depicted  on  each  participant’s  C  I  display. 
New  or  updated  enemy  locations  in  areas  not  currently  visible  on  their  tactical  display  might  go 
undetected.  Therefore,  the  CEP’s  C4I  instrumentation  was  designed  to  record  all  information 
depicted  on  the  participant’s  tactical  display,  and  the  precise  area  or  “window”  of  the  battlefield 
situation  visible  to  the  participant  at  any  time.  Assessing  visual  access  to  key  information 
updates,  such  as  priority  intelligence  requirements,  should  be  fairly  straightforward.  The 
instrumentation  can  automatically  track  the  exact  time  any  intelligence  update  is  first  available 
on  a  C4I  system,  and  any  delay  before  that  update’s  icon  becomes  visible  in  any  participant’s 
display  window. 

A  more  direct  measure  of  information  requirements  and  use  is  an  automated  log  of 
human-computer  interactions.  Such  logs  provide  a  relatively  complete  record  or  dialog  of  the 
information  exchanges  between  the  user  and  the  computer,  the  participant  and  the  C  I  system. 
Several  examples  of  how  this  log  information  might  provide  useful  instrumented  measures  are 
next  described,  based  on  the  instrumentation  developed  for  the  initial  CEP’s  C  I  systems. 

When  participants  wanted  more  detailed  information  on  troop  and  enemy  units,  they 
manually  accessed  and  called-up  this  information  to  their  C4I  displays.  The  log  recorded  the 
information  requested  (e.g.,  icon  selected)  and  provided  (report  content).  Related  log  measures 
include  manipulation  of  the  tactical  map  area  displayed  (e.g.,  pan,  scroll,  and  zoom)  and  terrain 
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analysis.  Instrumented  measures  from  such  log  data  can  help  identify  information  requirements 
and  assess  how  well  those  requirements  were  met. 

Direct  measures  of  information  requirements  are  possible  when  C4I  systems  allow  users 
to  preset  their  information  requests.  The  C4I  systems  for  future  CEPs  will  allow  participants  to 
specify  their  priority  information  requirements,  as  a  result  of  participants’  recommendations 
during  the  initial  evaluation.  The  (71  system  will  then  track  and  alert  the  user  when  the 
designated  activity  or  status  change  occurs.  A  log  of  related  human-computer  interactions, 
therefore,  should  provide  an  automated  set  of  participant  defined  information  requirements,  a 
record  of  how  and  when  these  requirements  were  met,  and  indicators  of  how  and  when  that 
information  was  used  based  on  participant  reaction  to  that  information. 

In  summary,  automated  logs  of  human  computer  interaction  on  C4I  systems  provide  a 
variety  of  instrumented  measures  for  identifying  and  assessing  information  requirements.  Logs 
on  information  requests  and  provisions,  in  particular,  provide  very  direct  measures.  Measures 
can  be  tabulated  at  the  individual  level  or  compiled  at  any  desired  collective  level.  Interactions 
are  automatically  time-stamped  to  the  simulated  test  scenario  to  determine  or  recreate  the  task 
context  at  the  time  of  occurrence. 


MDMP 


Results 


Since  there  was  no  comparison  (control)  group,  it  is  impossible  to  definitively  state 
whether  the  decision  making  process  in  the  CEP  was  more  effective  and  efficient  than  the 
current  MDMP.  However,  it  appeared  that  the  MDMP  used  in  the  CEP  was  marginally  more 
effective  and  efficient  than  the  current  MDMP  in  a  conventional  TOC.  No  steps  were  combined 
or  eliminated;  but  several  steps  were  performed  in  a  more  streamlined,  faster  and  less  formal 
manner  than  the  current  MDMP.  Reaction  to  battlefield  events  during  planning,  preparing  and 
execution  was  improved,  according  to  test  player  comments.  These  findings  are  not  unique. 
Elliot  et  al.  (1996)  report  similar  results  on  training  lessons  learned  from  the  Army  Warfighting 
Experiment  Focused  Dispatch.  During  Focused  Dispatch,  conventional  doctrinal  materials  for 
the  MDMP  were  available,  but  not  their  digital  counterparts.  The  Task  Force  personnel  in 
Focused  Dispatch,  as  in  the  CEP,  were  creating  new  decision  making  tactics,  techniques  and 
procedures  during  the  AWE.  In  the  absence  of  new  doctrine,  participants  fall  back  on  old 
doctrine.  Therefore,  the  lack  of  dramatic  change  in  the  MDMP  during  the  CEP  was  not 
surprising. 

Method  Improvements 

Approach.  Different  approaches  could  be  used  for  measuring  changes  in  the  decision 
making  process.  The  MDMP  is  the  doctrinally  approved  method  for  staffs  to  operate  under  in 
the  conventional  environment  (FM  101-5  [Department  of  the  Army,  1997]).  Since  the  issue  in 
this  research  concerned  changes  to  an  existing  decision  making  process  (MDMP),  the  MDMP 
was  used  as  the  baseline  and  changes  to  it  were  assessed.  Future  C4I  environments  will 
necessitate  commanders  to  operate  in  much  faster  response  cycles  (decision-execution 
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timeframes)  and  nearly  continuous  option  development  and  analysis  (Alberts,  1995).  The 
current,  cumbersome,  MDMP  is  almost  certainly  not  the  correct  doctrine  for  such  environments. 
Therefore  in  future  efforts  one  might  wish  to  be  more  open-ended  about  what  decision  making 
process  is,  or  should  be,  used. 

Manual  Measures.  Test  players  or  observers  could  simply  be  asked  to  list,  in  order,  steps 
the  staff  went  through  in  making  decisions.  Perhaps  methods  such  as  thinking  aloud  or 
structured  interviews  would  be  used  to  help  construct  the  decision  making  steps.  These  steps 
could  be  content-analyzed  with  the  goal  of  developing  a  reasonable  number  of  categories  (e.g., 

12  or  less,  perhaps  including  a  miscellaneous  category  for  low  frequency  responses).  For 
instance  responses  like  “determined  mission  requirements”  and  “analyzed  what  needed  to  be 
done  to  fulfill  higher  headquarters  mission”  would  each  be  categorized  as  “mission  analysis.” 
Then  each  response  could  be  assigned  to  one  of  the  categories.  Finally,  a  two-dimensional  chart 
could  be  developed,  with  step  number  (i.e.,  one  through  n)  as  the  rows  and  category  as  the 
columns.  Cells  would  contain  the  number  of  times  a  given  category  was  reported  for  a  given 
step.  Hopefully  one  process,  or  a  few  discrete  possible  decision  making  processes,  would 
emerge.  A  way  of  adding  some  structure  to  this  open-ended  method  would  be  to  ask  participants 
to  choose  among  existing  categories  or  steps.  In  addition  to  the  MDMP,  steps  from  research 
such  as  Klein  and  Crandall’s  (1996)  Recognition  Primed  Decision  (RPD)  model  and  Deckert, 
Entin,  Entin,  MacMillan  and  Serfaty’s  (1994)  work  on  expert  military  decision  makers  could  be 
provided  for  participants  or  observers  to  choose  from.  The  same  two-dimensional  table  of  step 
number  by  category  could  be  constructed.  Both  variants  have  the  advantage  of  allowing 
participants  to  describe  their  own  decision  making  process,  rather  than  confirm  or  reject  a 
process  provided  a  priori. 

Instrumented  Measures.  While  instrumented  measures  will  not  likely  yield  a  decision 
making  process,  they  could  be  useful  in  measuring  its  effectiveness  and  efficiency.  A  protocol 
could  be  designed  for  some  subject  matter  expert,  such  as  the  battle  master  to  enter  electronic 
“flags”  into  the  system  when  certain  events  occurred.  Some  possible  events  to  be  “flagged” 
could  be:  brigade  OPORD  issued;  battalion  OPORD  issued;  battalion  OPORD  execution  begun; 
brigade  fragmentary  order  (FRAGO)  1;  battalion  FRAGO  1;  enemy  COA  change  1;  battalion 
FRAGO  2.  Time  between  brigade  and  battalion  OPORD  reflects  planning  time.  Time  between 
battalion  OPORD  and  execution  reflects  preparation  time.  Time  between  brigade  and  battalion 
FRAGOs,  as  well  as  time  between  enemy  changing  COA  and  battalion  FRAGO  reflects  the 
“reaction  time”  of  the  staff  to  unanticipated  actions.  These  times  would  have  to  be  compared  to 
a  “standard”  or  to  those  of  a  control  group  to  operationally  define  changes  in  effectiveness  or 
efficiency  of  the  decision  making  process.  One  current  standard,  which  may  or  may  not  be 
applicable  to  future  staffs,  is  that  the  higher  headquarters  (e.g.,  battalion)  use  only  one  third  of 
time  available  for  planning,  leaving  two  thirds  of  the  available  time  for  the  subordinate  unit  (e.g., 
company  team). 


Summary 

The  methods  and  results  reported  for  this  initial  CEP  provide  a  preliminary  baseline  for 
assessing  information  requirements  and  military  decision  making  in  a  futuristic  operating 
environment.  The  results  are  in  no  way  definitive,  but  they  provide  a  benchmark  for  future  CEPs 
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and  related  efforts.  The  results  also  provide  many  valuable  suggestions  for  both  C4I  system  and 
evaluation  improvements.  Despite  CEP  design  and  execution  limitations,  the  value  of  the  results 
is  underscored  by  their  origin.  The  active  duty  CEP  test  players  were  immersed  for  two  weeks  in 
a  challenging  but  powerful  operational  environment.  The  CEP’s  battle  command  reorganization 
and  C4I  capabilities  provided  all  participants  and  the  Army  a  unique  opportunity  to  explore 
information  age  frontiers  and  glean  future  lessons  that  bear  on  current  issues. 

The  research  design  and  methods  used  during  the  CEP  were  limited,  but  provide  a  useful 
“strawman”  for  future  improvements  (Elliott  et  al.,  1998).  General  limitations  for  scenarios, 
training  and  test  design  were  noted  and  recommendations  made.  Specific  limitations  to  the 
manual  measures  developed  and  used  to  assess  information  requirements  and  military  decision 
making  were  also  described.  Recommendations  to  improve  these  measures  included:  a 
METT-T  structure  for  determining  information  requirements,  and  the  applicability  of  the  MDMP 
in  a  real-time  information  environment.  Improvement  recommendations  for  manual  measures 
were  also  considered  to  include  the  timing  and  scope  of  assessment  and  participants’  workload. 
Finally,  the  potential  of  instrumented  C4I  systems  to  reduce  participants’  workload  and  increase 
measurement  scope  and  precision  of  evaluation  was  encouraged  for  future  endeavors. 
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Appendix  A 


Information  Requirements  Questionnaire 


A-l 


PT#60-03A 


Insert  your  call  sign  in  this  space: 


Information  Requirements 


Purpose:  A  key  issue  with  respect  to  future  information  systems  is  how  well  they  meet  the 
information  requirements  of  the  commander  and  his  unit.  The  purpose  of  this  questionnaire  is  to 
help  address  that  issue  with  respect  to  your  duty  position  during  this  CEP. 

Instructions:  Consider  the  information  required  for  your  assigned  duty  position  during  the 
course  of  a  mission.  We  would  like  you  to  assess  how  well  the  information  systems  provided  at 
your  duty  station  during  this  CEP  met  the  information  requirements  of  your  duty  position. 

To  do  that,  we  ask  you  to  do  three  things: 

1.  Identify  the  types  of  information  required  in  your  CEP  duty  position. 

2.  Assess  how  well  each  identified  requirement  was  met  by  your  information  systems. 

3.  Recommend  how  your  information  requirements  might  be  better  met. 

A  sample  item  is  provided  below  to  illustrate  how  we  would  like  you  to  do  those  three  things. 


Not  Required 

Required 

Reqmt  Not  Met 

Reqmt  Met 

Reqmt  Exceeded 

Warning  Order 

Recommend: - 

□ 

2l 

□ 

21 

□ 

1.  First,  consider  the  type  of  information  listed  in  an  item,  such  as  Warning  Order  in  the 
above  sample.  If  that  information  is  not  a  requirement  for  your  duty  position,  check  Not 
Required  and  skip  to  the  next  item.  If  it  is  required,  check  Required,  as  in  the  above  sample. 

2.  If  you  check  Required,  then  assess  how  well  that  requirement  (Reqmt)  was  met.  To 
do  so,  check  either  Reqmt  Not  Met,  Reqmt  Met  (as  in  the  sample),  or  Reqmt  Exceeded. 

3.  If  you  have  any  recommendations  on  how  this  information  requirement  might  be 
better  met,  please  provide  them  on  the  Recommend  line.  If  you  have  no  recommendations,  skip 
to  the  next  item. 

This  questionnaire  identifies  some  types  of  information,  such  as  Warning  Order. 

What  other  information  requirements  need  to  be  met  for  your  duty  position? 

How  well  were  they  met? 

The  lower  portion  of  each  page  will  ask  you  to  identify  any  other  information  requirements,  as  in 
the  sample  item  below. 


Type  of  Information  Required 

Reqmt  Not  Met 

Reqmt  Met 

Reqmt  Exceeded 

Recommend: 

□ 

□ 

□ 

For  the  sample  item  above,  first  identify  in  writing  that  information  on  the  line  under 
Type  of  Information  Required.  Then,  as  for  the  Warning  Order  sample  item  above,  assess  how 
well  that  requirement  was  meet.  Finally,  provide  any  recommendations  to  better  meet  that 
requirement  on  the  Recommend  line. 

This  questionnaire  is  organized  by  Mission,  Enemy,  Terrain,  Troops  and  Time 
(METT-T).  Each  METT-T  factor  is  addressed  in  order  and  on  a  separate  page.  The  next  page, 
for  example,  is  titled  Mission  Related  Information  Requirements. 


Thank  you. 
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Mission  Related  Information  Requirements 
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Enemy  Related  Information  Requirements 
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Terrain  Related  Information  Requirements 


PT#60-03A 


Observation 

Not  Required 

□ 

Required 

□ 

Reqmt  Not  Met 

□ 

Reqmt  Met 

□ 

Reqmt  Exceeded 
□ 

Recommend:  — 

Cover 

Not  Required 

□ 

Required 

□ 

Reqmt  Not  Met 

□ 

Reqmt  Met 

□ 

Reqmt  Exceeded 

□ 

Recommend:  — 

Concealment 

Recommend:  — 

Not  Required 

□ 

Required 

□ 

Reqmt  Not  Met 

□ 

Reqmt  Met 
□ 

Reqmt  Exceeded 

□ 

Not  Required 

Required 

Reqmt  Not  Met 

Reqmt  Met 

Reqmt  Exceeded 

Obstacles 

□ 

□ 

□ 

□ 

□ 

Recommend:  — 

Key  Terrain 

Not  Required 
□ 

Required 

□ 

Reqmt  Not  Met 
□ 

Reqmt  Met 
□ 

Reqmt  Exceeded 

□ 

Recommend:  — 

Not  Required 

Required 

Reqmt  Not  Met 

Reqmt  Met 

Reqmt  Exceeded 

Approach  Avenues  CD 

□ 

□ 

□ 

□ 

Recommend: - 

Weather 

Not  Required 
□ 

Required 

□ 

Reqmt  Not  Met 

□ 

Reqmt  Met 

□ 

Reqmt  Exceeded 

□ 

Recommend:  — 

Are  there  any  other  types  of  Terrain  related  information  required  by  your  duty  position?  If  so,  please 
identify  the  type  of  information  required  below  and  then  assess  how  well  that  information  requirement 
was  met.  In  the  Recommend  section,  provide  your  recommendations  on  how  that  information 
requirement  might  be  better  met. 


Type  of  Information  Required 

Reqmt  Not  Met 
□ 

Reqmt  Met 
□ 

Reqmt  Exceeded 

□ 

Recommend: - 

Type  of  Information  Required 

Reqmt  Not  Met 

Reqmt  Met 

Reqmt  Exceeded 

□ 

□ 

□ 

Recommend: - 

i — - - _ — - - - - - —  - -  - -  — . . 

Type  of  Information  Required 

Reqmt  Not  Met 

Reqmt  Met 

Reqmt  Exceeded 

□ 

□ 

□ 

Recommend: - 
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Troop  Related  Information  Requirements 
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Time  Related  Information  Requirements 


Type  of  Information  Required  Reqmt  Not  Met  Reqmt  Met  Reqmt  Exceeded 

_  □  □  □ 


Recommend: 


Appendix  B 


Military  Decision  Making  Process  (MDMP)  Questionnaire 


B-l 


PT#60-03C 


CALL  SIGN 


MILITARY  DECISION  MAKING  PROCESS 


Mission 

(circle  one) 

Command  Post 
or  node 

(circle  one) 

Position 

(circle  one) 

MTC 

Command  Group 

Bn  Cdr 

Enemy  Ops  Off 

Defend 

Current  Ops 

XO 

Ops  SGT 

ATK 

Plans 

Battle  Captain 

Plans  SGT 

Exit  Questionnaire 

Floor  BCV 

Ops  Off 

Sensor  NCO 

- .  ==»:■■■  ■■  . .  :  . 

Friendly  Ops  Off 

OC 

Listed  below  are  the  seven  (7)  major  steps  of  the  Military  Decision  Making  Process  (MDMP),  for 
instance  used  in  tactical  decision  making.  Under  the  steps  the  sub-steps  involved  in  each  step  are 
also  listed.  Please  indicate,  by  checking  the  appropriate  blocks  whether  that  step  or  sub-step,  in  the 
restructured  Tactical  Operations  Center  (TOC),  was: 

UNCHANGED  -  The  entire  step  was  performed  the  same  way  in  the  restructured  TOC  as  in  a 
conventional  TOC.  If  you  check  “unchanged”  for  the  step,  do  not  check  any  blocks  for  the  sub-step 
under  that  step.  That  is,  if  the  entire  step  is  unchanged,  all  sub-steps  must  also  be  unchanged. 

CHANGED  -  The  step  or  sub-step  was  still  performed,  but  in  a  different  way  in  the  restructured  TOC 
than  in  a  conventional  TOC.  Under  the  comments  section,  please  describe  how  the  step  or  sub-step  was 
performed  differently.  Individual  sub-steps  under  the  step  may  be  unchanged,  changed,  combined  or 
eliminated. _ 

COMBINED  -  The  step  or  sub-step  was  performed  as  a  part  of  another  step  or  sub-step  in  the  MDMP  in 
the  restructured  TOC.  Under  the  comments  section,  indicate  which  other  step  or  sub-step  was  combined 
with  it.  Again,  sub-steps  under  the  step  may  be  unchanged,  changed,  combined  or  eliminated. 


ELIMINATED  -  The  entire  step  no  longer  needs  to  be  performed  at  all  in  the  restructured  TOC.  If  you 
check  “eliminated”  for  the  step,  do  not  check  any  blocks  for  the  sub-step  under  that  step.  That  is,  if  the 
entire  step  is  eliminated,  all  sub-steps  must  also  be  eliminated. 


UNCHANGED 

CHANGED 

COMBINED 

ELIMINATED 

MISSION  ANALYSIS 

-  Prepare  initial  Intelligence  Preparation  of 
the  Battlefield 

-  Analyze  higher  mission 

-  Analyze  Bde  order 

-  Assess  risk 

-  Briefing  and  approval 

Comments:  (required  if  changed  or  combined  are  checked)  Also,  please  include  the  addition  of 
any  related  new  steps  or  sub-steps. 
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UNCHANGED 

CHANGED 

COMBINED 

ELIMINATED 

COURSE  OF  ACTION  (COA) 
DEVELOPMENT 

-  Analyze  Cdr’s  guidance 

-  Analyze  relative  combat  power 

-  Generate  conceptual  possibilities 

-  Develop  scheme  of  maneuver 

-  Incorporate  all  battlefield  operating 
systems 

Comments:  (required  if  changed  or  combined  are  checked)  Also,  please  include  the  addition  of 
any  related  new  steps  or  sub-steps. 

UNCHANGED 

CHANGED 

COMBINED 

ELIMINATED 

WARGAMING 

-  Follow  wargaming  method 

-  Employ  all  friendly  and  enemy 
forces/resources 

-  Identify  critical  events  and  decision  points 

-  Record  results 

-  Apply  action/reaction/counteraction 

-  Assess  results 

Comments:  (required  if  changed  or  combined  are  checked)  Also,  please  include  the  addition  of 
any  related  new  steps  or  sub-steps. 
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_ _ _ UNCHANGED  CHANGED  COMBINED  ELIMINATED 

REHEARSAL _ 

-  Define  the  standard _ 

-  Orient  participants  to  terrain _ 

-  Verbalize  concept  of  the  operation _ 

-  Select  key  events  to  rehearse 

-  Describe  enemy  COAs  | 

-  Focus  on  purpose  of  rehearsal  (Decision 

Support  Template,  branches  &  Sequels, 
synchronization) _ 

Comments:  (required  if  changed  or  combined  are  circled)  Also,  please  include  the  addition  of 
any  related  new  steps  or  sub-steps. 


— — — - — - - — ■ — — — - - — - - * - - — — — - — - - - - - 1 

MISSION  EXECUTION  &  TRANSITION 

UNCHANGED 

_ 

CHANGED 

COMBINED 

ELIMINATED 

any  related  new  steps  or  sub-steps. 


PT#60-03C 


Rate  the  efficiency  and  effectiveness  with  which  each  of  the  seven  (7)  major  steps  of  the 
MDMP  were  performed  in  its  restructured  TOC  by  circling  the  appropriate  number.  If  a 
step  was  not  performed  (i.e.  eliminated)  leave  blank. 


MISSION  ANALYSIS 

Inefficient 

Moderately  Efficient 

Very  Efficient 

0 

1 

2 

Ineffective 

Moderately  Effective 

Very  Effective 

0 

1 

2 

COURSE  OF  ACTION  DEVELOPMENT 

Inefficient 

Moderately  Efficient 

Very  Efficient 

0 

1 

2 

Ineffective 

Moderately  Effective 

Very  Effective 

0 

1 

2 

WARGAMING 

Inefficient 

Moderately  Efficient 

Very  Efficient 

0 

1 

2 

Ineffective 

Moderately  Effective 

Very  Effective 

0 

1 

2 

COURSE  OF  ACTION  COMPARISON 

Inefficient 

Moderately  Efficient 

Very  Efficient 

0 

1 

2 

Ineffective 

Moderately  Effective 

Very  Effective 

0 

1 

2 
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ORDERS  PREPARATION 

Inefficient 

Moderately  Efficient 

Very  Efficient 

0 

1 

2 

Ineffective 

Moderately  Effective 

Very  Effective 

0 

1 

2 

REHEARSAL 

Inefficient 

Moderately  Efficient 

Very  Efficient 

0 

1 

2 

Ineffective 

Moderately  Effective 

Very  Effective 

0 

1 

2 

MISSION  EXECUTION  &  TRANSITION 

Inefficient 

Moderately  Efficient 

Very  Efficient 

0 

1 

2 

Ineffective 

Moderately  Effective 

Very  Effective 

0 

1 

2 
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Comments  on  Military  Decision  Making  Process  (MDMP) 
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Mission  Analysis 
(n  =  45) 


Comment 

Number 

More  streamlined  -  faster 

10 

Significantly  changed  due  to  technology 

10 

IPB  done  differently  because  of  tools 

6 

No  formal  briefing 

5 

Done  in  plans  vehicle 

3 

Performed  in  current  Ops 

3 

Must  consider  ADA  risk  for  UAV 

2 

XO  C2V  not  involved 

2 

More  terrain  analysis  than  last  mission 

1 

Analysis  individualized  due  to  technology 

1 

Current  Ops  executed  the  R  &  S  plan 

1 

Brigade  SITEMP  useless 

1 

Participated  via  voice  with  other  Operations  officers 

1 

Course  of  Action  Development 
(n  =  32) 


Comment 

Number 

Tools  provide  positive  capability  (including  updating  situation) 

7 

Must  have  maneuver-fire  support-intelligence  fusion  (i.e.;  too  few  BOS 
included) 

6 

S3  develops  CO  As 

3 

Struggled  to  develop  two  COAs 

2 

Relative  combat  power  to  force  on  force  ratios  not  performed 

2 

S2  develops  COAs 

2 

Should  be  quicker 

1 

Location  and  time  somewhat  changed 

1 

Rely  mostly  on  indirect  fire 

1 

COA  development  individualized 

1 

Did  not  have  capacity  to  do  it  right  this  time 

1 

COA  development  and  wargaming  combined 

1 

Add  BOS  at  rehearsal/wargaming 

1 

Commander’s  guidance  is  now  transparent 

1 

Is  much  quicker 

1 

Less  is  better  at  battalion  -  use  company  commanders  to  develop  COAs 

1 
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Wargaming 
(n  =  41) 


Comment 

Number 

Done  with  white  board  and  company  commanders 

7 

More  abridged 

6 

Results  recorded  in  white  board 

3 

S3  war  gamed 

2 

No  recording  or  assessing  of  results 

2 

No  specific  method  used 

2 

No  synchronization  matrix 

2 

Plans  C2V  issued  the  product 

2 

Combined  with  rehearsal 

2 

No  staff  huddle  with  synch  matrix 

1 

Need  assimilation  capability  that  allows  you  to  fight  the  plan 

1 

Only  key  events  wargamed 

1 

Done  by  teleconference  versus  terrain  model 

1 

Driven  by  commander 

1 

Plans  officers  all  drew  graphics  for  commander’s  CO  As 

1  1 

Steps  combined  with  COA  development 

1 

Fewer  people  involved 

1 

Excellent  -  solution  transparent 

1 

No  DST  consummated 

1 

Less  is  better  at  battalion  level 

1 

Flexible  schemes  can  replace  execution  of  a  plan 

1 

Other  BOS  not  incorporated 

1 

Course  of  Action  Comparison 
(n  =  35) 


Comment 

Number 

No  decision  matrix  used 

6 

Conducted  by  battalion  commander 

5 

Less  necessary  -  solution  transparent 

4 

Conducted  by  S3 _ 

3 

No  formal  criteria  used 

3 

Results  not  recorded 

2 

Did  not  weigh  criteria 

1 

Done  with  white  board 

1 

Decision  brief  eliminated 

1 

Demonstrates  the  value  of  COA  development 

1 

Get  verbal  feedback  on  COA 

1 

Must  still  stress  what  is  important  to  the  commander 

1 

Allows  Company  commanders  to  start  working  detailed  analysis  of  their 
requirements 

1 

Combined  with  wargaming 

1 

Eliminated 

1 

Use  of  plan  view  display  allowed  best  enemy  route  to  be  observed 

1 

Very  rudimentary  but  effective 

1 

Less  is  better  -  COA  can  change 

1 
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Orders  Preparation 
(n=19) 


Comment 

Number 

FRAGO  and  graphics  on  white  board  with  some  verbiage 

5 

Done  more  quickly 

3 

Largely  unchanged 

2 

Need  to  incorporate  all  BOS 

2 

Less  formal  -  event  driven 

2 

Done  by  S3 

1 

Does  not  need  detail,  just  key  BOS  required 

1 

OPORD  format  revised  due  to  fluid  exchange  of  information  and  increased 
situational  awareness 

1 

BOS  integrated  fully  at  rehearsal 

1 

Programmed  OPORD  format  would  speed  things  up 

1 

Rehearsal 
(n  =  41) 


Comments 

Number 

Conducted  with  company  commanders  via  white  board 

8 

No  formal  DST 

4 

Revised  plan  based  on  rehearsal  (Current  Ops) 

4 

Performed  via  FM  -  synchronization  drill,  not  sand  table 

2 

Not  focused  on  key  events 

2 

Better  rehearsal  this  time 

2 

Battle  Captain  acted  as  recorder 

1 

No  notional  staff  -  a  big  leap 

1 

Can  talk  to  DST  and  branches  if  needed  versus  ROC  drill 

1 

No  company  commander’s  brief  back 

1 

Direct  fire  distribution  not  rehearsed 

1 

Changed  only  in  aspect  of  medium  (teleconference) 

1 

May  be  performed  over  email 

1 

May  be  unnecessary 

1 

Looked  more  like  a  brief  by  battalion  commander 

1 

Combined  with  COA 

1 

All  pieces  required  were  present 

1 

The  commander  had  a  better  plan 

1 

Some  fire  support  questions  unanswered 

1 

Still  important 

1 

Players  can  make  this  more  effective 

1 

Still  meets  goals 

1 

XO  not  involved 

1 

Methods  and  tools  have  changed 

1 

Commander  is  key  person 

1 
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Mission  Execution 
(n  =  39) 


Comments 

Number 

Current  Ops  executed  the  R&S  plan 

6 

Greater  visualization  for  the  commander 

5 

Reaction  time  to  events  quicker 

3 

Less  spot  reports 

2 

2 

Must  use  this  technology,  think  “out  of  the  box” 

2 

Artillery  fired  from  C2Vs 

2 

More  efficient  &  effective 

2 

Fires  controlled  by  current  operations 

2 

No  calling  for  front  line  trace 

1 

Calls  for  fire  do  not  clog  radio 

1 

Fires  need  to  be  managed  by  commander’s  track 

1 

No  CSS  played 

1 

Continue  to  languish  with  doctrine  of  wait  until  conditions  are  perfect  to  LD 

1 

Must  deconflict  fires  and  close  air  support 

1 

Fires  and  routes  plan  synchronized  via  PVD 

1 

S3  moved  to  plans  track 

1 

Duties  of  plans  C2V  unclear  in  execution 

1 

Command  group  friendly  Ops  involved  in  CSS,  terrain  analysis 

1 

The  XO  monitored  current  Ops  and  synchronized  assets 

1 

Tools  used  by  commander  to  do  some  terrain  analysis 

1 

XO  synchronized  fires 

1 
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Appendix  D 
List  of  Acronyms 


D-l 


ABCS 

Army  Battle  Command  System 

ACR 

Armored  Cavalry  Regiment 

ADA 

Air  Defense  Artillery 

AFRU 

Armored  Forces  Research  Unit 

AOAC 

Armor  Officer  Advanced  Course 

AR 

assess  results 

ARI 

U.S.  Army  Research  Institute 

ASAS 

All  Source  Analysis  System 

ATCCS 

Army  Tactical  Command  and  Control  System 

AWE 

Army  Warfighting  Experiment 

BCV 

battle  command  vehicle 

Bde 

brigade 

BOS 

battlefield  operating  system 

C2V 

command  &  control  vehicle 

C4I 

command,  control,  communication,  computer,  and  intelligence 

CCIR 

commander’s  critical  information  requirements 

Cdr 

commander 

CEP 

concept  experimentation  program 

CGSC 

Command  and  General  Staff  College 

COA 

course  of  action 

CP 

command  post 

CPI 

Captain 

CSS 

combat  service  support 

CTCP 

combat  trains  command  post 

D 

dimensional 

DARPA 

Defense  Advanced  Research  Products  Agency 

DIS 

distributed  interactive  simulation 

DST 

decision  support  template 

E-MAIL 

electronic  mail 

FASTTRAIN 

Force  XXI  Training  Methods  and  Strategies 

FBC 

Future  Battlefield  Conditions 

FM 

field  manual 

FRAGO 

fragmentary  order 

IPB 

intelligence  preparation  of  the  battlefield 

ER 

information  requirement 

JSTARS 

Joint  Surveillance  Target  Attack  Radar  System 

MAJ 

Major 

MBBL 

Mounted  Battlespace  Battle  Lab 

MDMP 

military  decision  making  process 

ME&T 

mission  execution  and  transition 

METT-T 

mission,  enemy,  terrain,  troops,  and  time 

MEtT-TC 

mission,  enemy,  terrain,  troops,  time,  and  civilians 

ModSAF 

modular  semi -automated  forces 

MOA 

memorandum  of  agreement 

MOS 

military  occupational  specialty 

MWTB 

Mounted  Warfare  Test  Bed 
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NCO 

noncommissioned  officer 

NTC 

National  Training  Center 

0/C 

observer/controller 

OIC 

officer  in  charge 

OPFOR 

opposing  force 

OPORD 

operations  order 

Ops 

operations 

PVD 

plan  view  display 

RPD 

recognition  primed  decision 

RR 

record  results 

R&S 

reconnaissance  and  surveillance 

S2 

intelligence  officer 

S3 

operations  officer 

SFC 

Sergeant  First  Class 

SGT 

Sergeant 

SITEMP 

situation  template 

SITMAP 

situation  map 

SOP 

standing  operating  procedure 

SSG 

staff  sergeant 

TECO 

Test  and  Evaluation  Coordination  Office 

TOC 

tactical  operations  center 

TTP 

tactics,  techniques,  and  procedures 

UAV 

unmanned  aerial  vehicle 

USAARMC 

United  States  Army  Armor  Center 

VTC 

video  teleconference 

XO 

executive  officer 

